close#12692
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved error handling during migration rollbacks to better recognize and safely skip specific migration errors.
- Enhanced logging for migration rollback failures to provide clearer information without interrupting the process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Updated the default setting for the indexer feature to be disabled by default.
- Added a sample environment variable for enabling the indexer in the example configuration file.
- Introduced a new environment variable for the indexer in the CI workflow configuration.
- **Tests**
- Adjusted test configurations to explicitly enable the indexer feature during test execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
closes#12701
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved error handling during migration rollbacks to prevent unnecessary errors when migrations were never applied.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Improved log messages to include job IDs when jobs are removed from the queue, enhancing traceability for users monitoring job activity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-225
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added support for reading document blocks without requiring a workspace or root document snapshot.
- **Bug Fixes**
- Improved handling of large workspace snapshots by skipping them when they exceed 10MB.
- **Tests**
- Introduced new test cases to cover scenarios where root or workspace snapshots are absent.
- Expanded snapshot tests for document block reading.
- **Refactor**
- Updated several function signatures to make root and workspace snapshot parameters optional for greater flexibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Refactor**
- Adjusted logging verbosity to be more detailed in the 'canary' environment, providing debug-level logs, while maintaining info-level logs elsewhere.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Refactor**
- Improved visibility of job start and finish events by updating logging level, making these events more prominent in logs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Need to query the payload through job id for debugging
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Tests**
- Updated job metrics test to include an explicit job ID during execution.
- **Refactor**
- Enhanced job execution to support an optional job ID, improving job tracking and logging.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-223
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved error handling for search requests to prevent issues when error details are missing, ensuring clearer fallback messages for unknown errors.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved reliability of batch write operations to search providers, ensuring documents are correctly indexed and retrievable.
- **Tests**
- Added new test cases and snapshots to verify batch write functionality and confirm resolution of prior batch processing issues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Removed detailed verbose logging for Elasticsearch requests to reduce log noise.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close AF-2658
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Tests**
- Added new test cases and snapshots to enhance coverage for search results involving empty or missing fields like summary, title, and ref_doc_id.
- Verified consistent handling of empty string values and absence of fields across different search providers.
- **Bug Fixes**
- Improved handling of empty string values for specific fields by converting them to null to ensure consistent search result formatting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added '@affine/admin' to the list of selectable packages for development commands.
- **Enhancements**
- Improved package selection prompt by displaying up to 10 choices at a time for easier navigation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Implemented an exponential backoff retry strategy for job queues, resulting in progressively longer wait times between retry attempts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved system stability by ensuring that indexing jobs do not run when the indexer feature is disabled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
enable indexer using `compose.indexer.yml` on self-host:
```bash
docker compose -f compose.yml -f compose.indexer.yml up
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Updated the build process in the development container to include building the reader package.
- Added and configured a Manticore Search indexer service in the development container.
- Removed the indexer service and related environment variables from the self-hosted Docker Compose setup and environment example file.
- **Documentation**
- Improved documentation formatting for better readability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Updated workflow to revert changes to a specific i18n file after running the i18n build step during automated checks. No impact on user-facing features.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-221
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Resolved an issue that could cause the document listing process to enter an infinite loop under certain conditions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Tests**
- Refactored test setup for the database-backed document reader to improve modularity and clarity.
- Simplified database initialization and cleanup in tests.
- Updated mocks and assertions to align with the new test structure.
- Maintained existing test coverage and logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-220
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced a new service to handle indexing-related events and scheduled tasks, improving the management of document and workspace indexing.
- Added support for configuring the indexer feature via the AFFINE_INDEXER_ENABLED environment variable.
- **Bug Fixes**
- Ensured that indexing and deletion jobs are only enqueued when the indexer feature is enabled.
- **Tests**
- Added comprehensive tests for the new indexing event service, covering various configuration scenarios.
- Removed obsolete test related to auto-indexing scheduling.
- **Chores**
- Updated configuration descriptions and mappings to improve clarity and environment variable support.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added a configurable batch size setting for automatic workspace indexing. Users can now adjust how many workspaces are indexed per batch, with a default value of 10 and support for values between 1 and 1000.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
CLOUD-214
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Improved search functionality to support stemming, allowing searches for variations of words (e.g., "window", "windows", "design") to return relevant results.
- **Tests**
- Added new tests to verify that search results correctly highlight and match stemmed word variations in document titles.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-207
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced automated periodic indexing of workspaces with a new job type and a scheduled cron job running every 30 seconds.
- Added a unique sequential identifier (`sid`) and an "indexed" flag to workspaces to track indexing status.
- **Improvements**
- Enhanced workspace indexing to handle missing workspaces and snapshots distinctly and selectively index documents.
- Added ability to query workspaces after a given identifier with result limits.
- **Bug Fixes**
- Improved error handling and logging during workspace indexing operations.
- **Tests**
- Expanded test coverage for workspace indexing and auto-indexing, including scheduling and edge cases.
- **Chores**
- Updated data models and schema to support new workspace fields and indexing features.
- Enhanced mock data utilities to allow custom timestamps.
- Improved type safety and flexibility in document snapshot retrieval.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-208
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Enhanced document access control with batch permission checks, enabling efficient filtering of documents based on user roles and permissions.
- Added detailed document-level role and permission management for workspace users.
- **Bug Fixes**
- Improved accuracy in filtering search results to only display documents users have permission to read.
- **Tests**
- Added comprehensive tests for document-level permission filtering and search result accuracy.
- Introduced new mock utilities to support permission-related test scenarios.
- **Refactor**
- Simplified and optimized permission logic for determining user roles and document access.
- **Documentation**
- Updated type definitions for improved clarity in permission handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Added a `.gitignore` file to prevent the `.env` file in the self-hosted Docker directory from being tracked by Git.
- **New Features**
- Introduced a new environment variable for the search endpoint in the Docker Compose configuration for improved service connectivity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-217
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced a dedicated test job for Elasticsearch, running only Elasticsearch-specific tests during CI.
- **Chores**
- Enhanced server test workflows with explicit setup steps and automated coverage uploads.
- Improved test suite structure for Elasticsearch provider to enable conditional and asynchronous test execution based on environment variables.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Standardized the naming of the search service to "indexer" in automated test workflows.
- Ensured the "indexer" service is available in additional test jobs for improved consistency across workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-137
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced advanced workspace-scoped search and aggregation capabilities with support for complex queries, highlights, and pagination.
- Added pluggable search providers: Elasticsearch and Manticoresearch.
- New GraphQL queries, schema types, and resolver support for search and aggregation.
- Enhanced configuration options for search providers in self-hosted and cloud deployments.
- Added Docker Compose services and environment variables for Elasticsearch and Manticoresearch.
- Integrated indexer service into deployment and CI workflows.
- **Bug Fixes**
- Improved error handling with new user-friendly error messages for search provider and indexer issues.
- **Documentation**
- Updated configuration examples and environment variable references for indexer and search providers.
- **Tests**
- Added extensive end-to-end and provider-specific tests covering indexing, searching, aggregation, deletion, and error cases.
- Included snapshot tests and test fixtures for search providers.
- **Chores**
- Updated deployment scripts, Helm charts, and Kubernetes manifests to include indexer-related environment variables and secrets.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Corrected CPU resource allocation for the documentation service to ensure accurate deployment configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Updated a backend dependency to the latest version to ensure continued compatibility and improvements. No user-facing changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fengmk2
