<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Tests**
- Refactored test setup for the database-backed document reader to improve modularity and clarity.
- Simplified database initialization and cleanup in tests.
- Updated mocks and assertions to align with the new test structure.
- Maintained existing test coverage and logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-220
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced a new service to handle indexing-related events and scheduled tasks, improving the management of document and workspace indexing.
- Added support for configuring the indexer feature via the AFFINE_INDEXER_ENABLED environment variable.
- **Bug Fixes**
- Ensured that indexing and deletion jobs are only enqueued when the indexer feature is enabled.
- **Tests**
- Added comprehensive tests for the new indexing event service, covering various configuration scenarios.
- Removed obsolete test related to auto-indexing scheduling.
- **Chores**
- Updated configuration descriptions and mappings to improve clarity and environment variable support.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added a configurable batch size setting for automatic workspace indexing. Users can now adjust how many workspaces are indexed per batch, with a default value of 10 and support for values between 1 and 1000.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
CLOUD-214
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Improved search functionality to support stemming, allowing searches for variations of words (e.g., "window", "windows", "design") to return relevant results.
- **Tests**
- Added new tests to verify that search results correctly highlight and match stemmed word variations in document titles.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-207
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced automated periodic indexing of workspaces with a new job type and a scheduled cron job running every 30 seconds.
- Added a unique sequential identifier (`sid`) and an "indexed" flag to workspaces to track indexing status.
- **Improvements**
- Enhanced workspace indexing to handle missing workspaces and snapshots distinctly and selectively index documents.
- Added ability to query workspaces after a given identifier with result limits.
- **Bug Fixes**
- Improved error handling and logging during workspace indexing operations.
- **Tests**
- Expanded test coverage for workspace indexing and auto-indexing, including scheduling and edge cases.
- **Chores**
- Updated data models and schema to support new workspace fields and indexing features.
- Enhanced mock data utilities to allow custom timestamps.
- Improved type safety and flexibility in document snapshot retrieval.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-208
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Enhanced document access control with batch permission checks, enabling efficient filtering of documents based on user roles and permissions.
- Added detailed document-level role and permission management for workspace users.
- **Bug Fixes**
- Improved accuracy in filtering search results to only display documents users have permission to read.
- **Tests**
- Added comprehensive tests for document-level permission filtering and search result accuracy.
- Introduced new mock utilities to support permission-related test scenarios.
- **Refactor**
- Simplified and optimized permission logic for determining user roles and document access.
- **Documentation**
- Updated type definitions for improved clarity in permission handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Added a `.gitignore` file to prevent the `.env` file in the self-hosted Docker directory from being tracked by Git.
- **New Features**
- Introduced a new environment variable for the search endpoint in the Docker Compose configuration for improved service connectivity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-217
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced a dedicated test job for Elasticsearch, running only Elasticsearch-specific tests during CI.
- **Chores**
- Enhanced server test workflows with explicit setup steps and automated coverage uploads.
- Improved test suite structure for Elasticsearch provider to enable conditional and asynchronous test execution based on environment variables.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Standardized the naming of the search service to "indexer" in automated test workflows.
- Ensured the "indexer" service is available in additional test jobs for improved consistency across workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-137
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced advanced workspace-scoped search and aggregation capabilities with support for complex queries, highlights, and pagination.
- Added pluggable search providers: Elasticsearch and Manticoresearch.
- New GraphQL queries, schema types, and resolver support for search and aggregation.
- Enhanced configuration options for search providers in self-hosted and cloud deployments.
- Added Docker Compose services and environment variables for Elasticsearch and Manticoresearch.
- Integrated indexer service into deployment and CI workflows.
- **Bug Fixes**
- Improved error handling with new user-friendly error messages for search provider and indexer issues.
- **Documentation**
- Updated configuration examples and environment variable references for indexer and search providers.
- **Tests**
- Added extensive end-to-end and provider-specific tests covering indexing, searching, aggregation, deletion, and error cases.
- Included snapshot tests and test fixtures for search providers.
- **Chores**
- Updated deployment scripts, Helm charts, and Kubernetes manifests to include indexer-related environment variables and secrets.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Corrected CPU resource allocation for the documentation service to ensure accurate deployment configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Updated a backend dependency to the latest version to ensure continued compatibility and improvements. No user-facing changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fengmk2
