fix AI-392
fix AI-393
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Improved filtering of outdated document ID styles in embedding status
reporting, ensuring more accurate counts of embedded documents.
* Stricter rate limiting applied to workspace embedding status queries
for enhanced system reliability.
* **Bug Fixes**
* Resolved issues with duplicate or outdated document IDs affecting
embedding status totals.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-359
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added automated cleanup of embeddings for documents deleted or trashed
from workspaces.
* Introduced a new job to schedule and perform this cleanup per
workspace daily and on demand.
* Added new GraphQL mutation to manually trigger the cleanup process.
* Added the ability to list workspaces with flexible filtering and
selection options.
* **Improvements**
* Enhanced document status handling to more accurately reflect embedding
presence.
* Refined internal methods for managing and checking document
embeddings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-367
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved document filtering to exclude settings documents and empty
blobs from embedding and status calculations.
* Enhanced embedding jobs to skip processing deprecated documents if a
newer version exists, ensuring only up-to-date documents are embedded.
* **New Features**
* Added a mutation to trigger the cron job for generating missing
titles.
* **Tests**
* Added test to verify exclusion of documents with empty content from
embedding.
* Updated embedding-related tests to toggle embedding state during
attachment upload under simulated network conditions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-350
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a new option to manually trigger the generation of missing
session titles via a GraphQL query.
* **Improvements**
* The process for generating missing session titles now considers all
eligible sessions, without limiting the number processed at a time.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close AF-2732
#### PR Dependency Tree
* **PR #13139** 👈
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for a document summary field, allowing documents to
include and display an optional summary alongside the title.
* **Bug Fixes**
* Improved access control when retrieving documents, ensuring proper
permission checks are enforced.
* **Tests**
* Expanded test coverage to verify correct handling of document title
and summary fields, including cases where the summary is absent.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved session filtering to use the correct criteria when querying
sessions, ensuring more accurate results based on the action parameter.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced new API endpoints and GraphQL queries to retrieve Copilot
chat sessions by workspace, document, and pinned status, with detailed
session and message information.
* Added support for filtering and querying Copilot chat histories with
new options such as pinned status and message ordering.
* **Bug Fixes**
* Improved filtering logic for listing and retrieving chat sessions,
ensuring accurate results for workspace, document, and pinned session
queries.
* **Tests**
* Expanded and refactored test coverage for session listing, filtering,
and new query options to ensure reliability and correctness of Copilot
session retrieval.
* Updated snapshot data to reflect new session types and filtering
capabilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved validation to prevent updates to sessions with action prompts
and restrict certain updates on forked sessions.
- **Tests**
- Expanded and clarified test coverage for session updates, pinning
behavior, and session type conversions, with more explicit error
handling and validation scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved handling of document statuses to ensure documents without a finished or existing status are now explicitly marked as "processing" instead of remaining undefined.
- **Tests**
- Added comprehensive new tests and snapshot entries to verify document status merging, including edge cases and concurrent operations, ensuring robust and consistent behavior.
- **Enhancements**
- Updated context document listings to display the processing status for relevant documents.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Improvements**
- Increased the default concurrency for background tasks, enhancing processing efficiency.
- Improved handling of empty or unsupported documents to ensure consistent processing.
- Optimized document filtering to exclude certain documents from processing, improving performance.
- **Bug Fixes**
- Enhanced detection of empty document summaries, reducing errors during processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-10
fix AI-109
fix PD-2484
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added a method to check if a document requires embedding, improving embedding efficiency.
- Enhanced document embeddings with enriched metadata, including title, summary, creation/update dates, and author information.
- Introduced a new type for document fragments with extended metadata fields.
- **Improvements**
- Embedding logic now conditionally processes only documents needing updates.
- Embedding content now includes document metadata for more informative context.
- Expanded and improved test coverage for embedding scenarios and workspace behaviors.
- Event emission added for workspace embedding updates on client version mismatch.
- Job queueing enhanced with prioritization and explicit job IDs for better management.
- Job queue calls updated to include priority and context identifiers in a structured format.
- **Bug Fixes**
- Improved handling of ignored documents in embedding matches.
- Fixed incorrect document ID assignment in embedding job queueing.
- **Tests**
- Added and updated snapshot and behavioral tests for embedding and workspace document handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-127
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added automated event handling for workspace updates and document embedding, streamlining document embedding workflows.
- Introduced detection and queuing of documents needing embedding, excluding ignored documents.
- **Improvements**
- Enhanced performance of embedding-related searches by filtering results at the database level.
- Increased concurrency for embedding job processing to improve throughput.
- **Bug Fixes**
- Improved error handling and fallback for missing document titles during embedding.
- Added safeguards to skip invalid embedding jobs based on document identifiers.
- **Tests**
- Expanded test coverage for document embedding and ignored document filtering.
- Updated end-to-end tests to use dynamic content for improved reliability.
- Added synchronization waits in document creation utilities to improve test stability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved the accuracy of document matching by excluding ignored documents from search results.
- **Chores**
- Updated internal handling of ignored document lists for better consistency and reliability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-20
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Enhanced file metadata with MIME type, blob ID, and file name across context and workspace, now visible in UI and API.
- Added workspace-level matching for files and documents with configurable thresholds and workspace scoping in search queries.
- Introduced a new error type and user-friendly messaging for global workspace context matching failures.
- **Bug Fixes**
- Improved consistent handling of file MIME types and nullable context IDs for accurate metadata.
- **Documentation**
- Updated GraphQL schema, queries, and mutations to include new metadata fields, optional parameters, and error types.
- **Style**
- Added new localization strings for global context matching error messages.
- **Tests**
- Extended test coverage with new and updated snapshot tests for metadata and matching logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added paginated document listing for workspaces, allowing users to browse documents with pagination controls.
- Enhanced document details to display creation and update timestamps, as well as information about the creator and last updater.
- **Bug Fixes**
- Updated deprecation notice for workspace document metadata fields to guide users to the latest recommended field.
- **Tests**
- Added new tests to verify document info retrieval and pagination functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Expanded file chunk matching to include both context and workspace file embeddings, providing broader and more relevant search results.
- **Improvements**
- Enhanced result ranking by introducing a re-ranking step for combined embedding matches, improving the relevance of returned file chunks.
- Adjusted file count reporting to reflect the total number of workspace files instead of ignored documents for more accurate workspace file statistics.
- Renamed and streamlined workspace file management methods for clearer and more consistent API usage.
- **Bug Fixes**
- Prevented embedding similarity queries when embedding is disabled for a workspace, improving system behavior consistency.
- **Tests**
- Added comprehensive tests to verify workspace embedding management, including enabling, matching, and disabling embedding functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
DarkSky
