<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Improved internal monitoring for AI embedding operations to enhance reliability and performance tracking. No changes to user-facing features.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved handling of workspace embedding events to ensure they are only triggered for workspace-type spaces.
- **Chores**
- Added additional debug logging for document embedding jobs to aid in monitoring and troubleshooting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Improvements**
- Increased the default concurrency for background tasks, enhancing processing efficiency.
- Improved handling of empty or unsupported documents to ensure consistent processing.
- Optimized document filtering to exclude certain documents from processing, improving performance.
- **Bug Fixes**
- Enhanced detection of empty document summaries, reducing errors during processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved reliability by ensuring certain features are only enabled when required support and configuration are present, reducing the risk of runtime errors.
- Enhanced platform detection logic for better accuracy across different environments, including macOS and Windows systems.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-10
fix AI-109
fix PD-2484
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added a method to check if a document requires embedding, improving embedding efficiency.
- Enhanced document embeddings with enriched metadata, including title, summary, creation/update dates, and author information.
- Introduced a new type for document fragments with extended metadata fields.
- **Improvements**
- Embedding logic now conditionally processes only documents needing updates.
- Embedding content now includes document metadata for more informative context.
- Expanded and improved test coverage for embedding scenarios and workspace behaviors.
- Event emission added for workspace embedding updates on client version mismatch.
- Job queueing enhanced with prioritization and explicit job IDs for better management.
- Job queue calls updated to include priority and context identifiers in a structured format.
- **Bug Fixes**
- Improved handling of ignored documents in embedding matches.
- Fixed incorrect document ID assignment in embedding job queueing.
- **Tests**
- Added and updated snapshot and behavioral tests for embedding and workspace document handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-4
fix AI-18
better provider/model choose to allow fallback to similar models (e.g., self-hosted) when the provider is not fully configured
split functions of different output types
fix CLOUD-219
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Ensured that documents without content now receive a placeholder embedding, improving consistency in document processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-127
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added automated event handling for workspace updates and document embedding, streamlining document embedding workflows.
- Introduced detection and queuing of documents needing embedding, excluding ignored documents.
- **Improvements**
- Enhanced performance of embedding-related searches by filtering results at the database level.
- Increased concurrency for embedding job processing to improve throughput.
- **Bug Fixes**
- Improved error handling and fallback for missing document titles during embedding.
- Added safeguards to skip invalid embedding jobs based on document identifiers.
- **Tests**
- Expanded test coverage for document embedding and ignored document filtering.
- Updated end-to-end tests to use dynamic content for improved reliability.
- Added synchronization waits in document creation utilities to improve test stability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Expanded file chunk matching to include both context and workspace file embeddings, providing broader and more relevant search results.
- **Improvements**
- Enhanced result ranking by introducing a re-ranking step for combined embedding matches, improving the relevance of returned file chunks.
- Adjusted file count reporting to reflect the total number of workspace files instead of ignored documents for more accurate workspace file statistics.
- Renamed and streamlined workspace file management methods for clearer and more consistent API usage.
- **Bug Fixes**
- Prevented embedding similarity queries when embedding is disabled for a workspace, improving system behavior consistency.
- **Tests**
- Added comprehensive tests to verify workspace embedding management, including enabling, matching, and disabling embedding functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
