<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Smarter embedding pipeline skips re-embedding when content hasn’t
changed; added content sanitization for embeddings and workspace content
retrieval.
- Bug Fixes
- Re-embedding now requires both a document update and the last
embedding being older than 10 minutes, reducing unnecessary work.
- Refactor
- Consolidated embedding preprocessing and moved sanitization utilities
into shared models; upserts now refresh stored content.
- Tests
- Expanded snapshot-based tests covering multiple time/age scenarios for
embedding decision logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Prevents accidental deletion of placeholder documents during embedding
cleanup.
* Improves accuracy when identifying documents to remove, using multiple
data sources.
* Skips unnecessary cleanup when no embeddings or snapshots exist,
reducing noise and overhead.
* **Chores**
* Streamlined and centralized document filtering logic to ensure
consistent cleanup behavior.
* Parallelized data checks to make cleanup more efficient without
changing user workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- None
- Bug Fixes
- Ensures a placeholder embedding is always created when content is
empty or after deletion, reducing errors and improving Copilot
stability.
- Refactor
- Centralized empty-embedding handling for consistent behavior across
workflows.
- Standardized embedding dimension configuration to a single source for
reliability.
- Chores
- Simplified internal embedding module surface and imports for
maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-408
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a new field to workspaces to track the last time embeddings were
checked.
* Cleanup jobs for workspace embeddings now skip workspaces that haven't
changed in over 30 days or have no embeddings, improving efficiency.
* Cleanup jobs are now automatically triggered when a workspace is
updated.
* **Improvements**
* Enhanced workspace selection for cleanup and indexing tasks to use
more precise filters and batching.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Updated scenario names and options for Copilot, including new and
renamed scenarios such as "audio_transcribing,"
"complex_text_generation," "quick_decision_making,"
"quick_text_generation," and "polish_and_summarize."
* Enhanced support for customizing and overriding default model
assignments in Copilot scenarios.
* **Bug Fixes**
* Improved consistency and clarity in scenario configuration and prompt
selection.
* **Documentation**
* Updated descriptions in configuration interfaces to better explain the
ability to use custom models and override defaults.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-404
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced scenario-based configuration for copilot, allowing default
model assignments for various AI use cases.
* Added a new image generation model to the available options.
* **Improvements**
* Refined copilot provider settings by removing deprecated fallback
options and standardizing base URL configuration.
* Enhanced prompt management to support scenario-driven updates and
improved configuration handling.
* Updated admin and settings interfaces to support new scenario
configurations.
* **Bug Fixes**
* Removed deprecated or unused prompts and related references across
platforms for consistency.
* **Other**
* Improved test coverage and updated test assets to reflect prompt and
scenario changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved permission checks when adding context blobs to ensure only
authorized users can perform this action.
* **Refactor**
* Streamlined background processing of blob embeddings by removing
user-specific parameters and simplifying job handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for managing "blobs" in Copilot context, including
adding and removing blobs via new GraphQL mutations and UI fields.
* Introduced tracking and querying of blob embeddings within workspaces,
enabling search and similarity matching for blob content.
* Extended Copilot context and workspace APIs, schema, and UI to display
and manage blobs alongside existing documents and files.
* **Bug Fixes**
* Updated context and embedding status logic to handle blobs, ensuring
accurate status reporting and embedding management.
* **Tests**
* Added and updated test cases and snapshots to cover blob embedding
insertion, matching, and removal scenarios.
* **Documentation**
* Updated GraphQL schema and TypeScript types to reflect new
blob-related fields and mutations.
* **Chores**
* Refactored and cleaned up code to support new blob entity and
embedding logic, including renaming and updating internal methods and
types.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-359
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added automated cleanup of embeddings for documents deleted or trashed
from workspaces.
* Introduced a new job to schedule and perform this cleanup per
workspace daily and on demand.
* Added new GraphQL mutation to manually trigger the cleanup process.
* Added the ability to list workspaces with flexible filtering and
selection options.
* **Improvements**
* Enhanced document status handling to more accurately reflect embedding
presence.
* Refined internal methods for managing and checking document
embeddings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-367
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved document filtering to exclude settings documents and empty
blobs from embedding and status calculations.
* Enhanced embedding jobs to skip processing deprecated documents if a
newer version exists, ensuring only up-to-date documents are embedded.
* **New Features**
* Added a mutation to trigger the cron job for generating missing
titles.
* **Tests**
* Added test to verify exclusion of documents with empty content from
embedding.
* Updated embedding-related tests to toggle embedding state during
attachment upload under simulated network conditions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Error messages for unavailable copilot providers now include specific
model IDs for clearer context.
* Added new detailed error messages for embedding generation failures
specifying provider and error details.
* The API and GraphQL schema have been extended with new error types
reflecting these detailed error cases.
* **Bug Fixes**
* Enhanced error handling to detect and report incomplete or missing
embeddings from providers.
* Added safeguards to skip embedding insertions when no embeddings are
provided, preventing unnecessary processing.
* **Documentation**
* Updated localization and translation keys to support dynamic error
messages with model IDs and provider details.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Improved document reranking with a more streamlined and accurate
scoring system.
* Enhanced support for binary ("yes"/"no") document relevance judgments.
* **Improvements**
* Simplified user prompts and output formats for reranking tasks, making
results easier to interpret.
* Increased reliability and consistency in document ranking results.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-231
#### PR Dependency Tree
* **PR #12893** 👈
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Updated background processing for document indexing and deletion to
use a job queue system instead of event-based triggers.
- **Bug Fixes**
- Improved reliability of embedding updates and deletions by ensuring
tasks are properly queued and processed.
- **Tests**
- Adjusted tests to verify that document operations correctly trigger
job queue actions.
No changes to user-facing features or interface.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#### PR Dependency Tree
* **PR #12871** 👈
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Updated internal logging mechanism to use a standard logger, resulting
in a simplified and more consistent logging approach. No changes to
user-facing features or functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#### PR Dependency Tree
* **PR #12871** 👈
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
#### PR Dependency Tree
* **PR #12857** 👈
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved error handling in document embedding jobs to log and skip
errors instead of interrupting the process.
- Enhanced logging with clearer messages about embedding outcomes and
skipped operations.
- Added warnings when documents or summaries are missing during
embedding.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
DarkSky