fix AF-2748
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Improved mail sending job with adaptive retry delays based on elapsed
time, enhancing reliability of email delivery.
* **Chores**
* Updated job payload to include a start time for better retry
management.
* Added an internal delay utility to support asynchronous pause in
processes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Error messages for unavailable copilot providers now include specific
model IDs for clearer context.
* Added new detailed error messages for embedding generation failures
specifying provider and error details.
* The API and GraphQL schema have been extended with new error types
reflecting these detailed error cases.
* **Bug Fixes**
* Enhanced error handling to detect and report incomplete or missing
embeddings from providers.
* Added safeguards to skip embedding insertions when no embeddings are
provided, preventing unnecessary processing.
* **Documentation**
* Updated localization and translation keys to support dynamic error
messages with model IDs and provider details.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-235
<img width="2104" height="1200" alt="image"
src="https://github.com/user-attachments/assets/6ea0fd89-ab32-42e3-a675-f00f9e5856ad"
/>
#### PR Dependency Tree
* **PR #13148** 👈
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced a new monitoring service that automatically collects and
logs process memory usage statistics every minute.
* Enhanced system monitoring capabilities by integrating a global
monitoring module.
* Added support for a new "process" scope in metric tracking.
* **Chores**
* Improved internal module organization by including the monitoring
module in the core functionality set.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close AF-2728
#### PR Dependency Tree
* **PR #13143** 👈
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added metrics tracking for comment attachment uploads, including
recording attachment size and total uploads by MIME type.
* Enhanced logging for attachment uploads with detailed information such
as workspace ID, document ID, file size, and user ID.
* **Chores**
* Expanded internal metric categories to include storage-related
metrics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close AF-2705
#### PR Dependency Tree
* **PR #12984** 👈
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* The Apple OAuth provider is now available only for clients version
0.22.0 or higher.
* Client version detection has been improved by extracting version
information from request headers.
* **Bug Fixes**
* Ensured that the Apple OAuth provider is hidden for clients below
version 0.22.0.
* **Tests**
* Added comprehensive end-to-end and utility tests for OAuth provider
selection and client version extraction.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-227
close CLOUD-230
#### PR Dependency Tree
* **PR #12761** 👈
* **PR #12924**
* **PR #12925**
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced a comprehensive commenting system, enabling users to
create, update, resolve, and delete comments and replies on documents.
* Added support for uploading attachments to comments, with clear error
messaging if size limits are exceeded.
* Implemented role-based permissions for comment actions, including a
new "Commenter" role.
* Enabled paginated listing and change tracking of comments and replies
via GraphQL queries.
* Provided full localization and error handling for comment-related
actions.
* **Bug Fixes**
* Improved uniqueness handling when fetching user data for comments and
replies.
* **Documentation**
* Extended GraphQL schema and frontend localization to document and
support new comment features.
* **Tests**
* Added extensive backend test suites covering all comment and reply
functionalities, permissions, and attachment uploads.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-230
#### PR Dependency Tree
* **PR #12911** 👈
* **PR #12761**
* **PR #12924**
* **PR #12925**
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for uploading and retrieving comment attachments in
workspace documents via a new API endpoint.
* Introduced a service for managing comment attachments, including
storage, retrieval, deletion, and URL generation.
* Implemented localized error messages and improved error handling for
missing comment attachments.
* **Bug Fixes**
* Improved error feedback when comment attachments are not found.
* **Tests**
* Added comprehensive tests for comment attachment storage, retrieval,
deletion, API endpoint behavior, and permission checks.
* **Documentation**
* Updated GraphQL schema and localization files to include new error
types for comment attachments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-226
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced support for comments and replies within workspaces and
documents, enabling users to create, update, delete, and resolve
comments, as well as manage threaded replies.
- **Bug Fixes**
- Added user-friendly error messages and handling for situations where
comments or replies are not found.
- **Tests**
- Added comprehensive tests to ensure correct behavior of comment and
reply operations.
- **Localization**
- Added English translations for new comment and reply error messages.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#### PR Dependency Tree
* **PR #12760** 👈
* **PR #12909**
* **PR #12911**
* **PR #12761**
* **PR #12924**
* **PR #12925**
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
close CLOUD-233
#### PR Dependency Tree
* **PR #12950** 👈
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added support for configuring multiple server hosts across backend and
frontend settings.
* Enhanced deployment and Helm chart configuration to allow specifying
multiple ingress hosts.
* Updated admin and configuration interfaces to display and manage
multiple server hosts.
* **Improvements**
* Improved URL generation, OAuth, and worker service logic to
dynamically handle requests from multiple hosts.
* Enhanced captcha verification to support multiple allowed hostnames.
* Updated frontend logic for platform-specific server base URLs and
allowed origins, including Apple app domains.
* Expanded test coverage for multi-host scenarios.
* **Bug Fixes**
* Corrected backend logic to consistently use dynamic base URLs and
origins based on request host context.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix android & ios selfhost syncing
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved WebSocket connection compatibility by enabling cross-origin
requests with credentials and restricting allowed HTTP methods to GET
and POST.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: fengmk2 <fengmk2@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced support for parsing and converting duration strings (e.g., "1h30m") into milliseconds and seconds.
- Added utility methods to handle a wide range of time units and their combinations.
- Added functions to calculate dates offset before or after a given date by specified durations.
- **Tests**
- Implemented comprehensive automated tests to ensure accurate parsing and conversion of duration strings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Improved log messages to include job IDs when jobs are removed from the queue, enhancing traceability for users monitoring job activity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Refactor**
- Improved visibility of job start and finish events by updating logging level, making these events more prominent in logs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Need to query the payload through job id for debugging
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Tests**
- Updated job metrics test to include an explicit job ID during execution.
- **Refactor**
- Enhanced job execution to support an optional job ID, improving job tracking and logging.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Improvements**
- Increased the default concurrency for background tasks, enhancing processing efficiency.
- Improved handling of empty or unsupported documents to ensure consistent processing.
- Optimized document filtering to exclude certain documents from processing, improving performance.
- **Bug Fixes**
- Enhanced detection of empty document summaries, reducing errors during processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added support for a new AES key for license management, improving license encryption and decryption processes.
- **Bug Fixes**
- Improved error messages and handling when activating expired or invalid licenses.
- **Refactor**
- Updated license decryption logic to use a fixed AES key instead of deriving one from the workspace ID.
- Added validation for environment variable values to prevent invalid configurations.
- **Tests**
- Enhanced license-related tests to cover new key usage and updated error messages.
- Updated environment variable validation tests with clearer error messages.
- **Chores**
- Updated environment variable handling for improved consistency.
- Set production environment variable explicitly in build configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added a GraphQL mutation to validate multiple app configuration updates, returning detailed validation results for each item.
- Extended the API schema to support validation feedback, enabling client-side checks before applying changes.
- Introduced a detailed, parameterized error message system for configuration validation errors.
- Enabled validation of configuration inputs via the admin UI with clear, descriptive error messages.
- **Improvements**
- Enhanced error reporting with specific, context-rich messages for invalid app configurations.
- Simplified admin settings UI by removing the confirmation dialog and streamlining save actions.
- Improved clarity and maintainability of validation logic and error handling components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Implemented an exponential backoff retry strategy for job queues, resulting in progressively longer wait times between retry attempts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Users are now prevented from deleting their account if they own one or more team workspaces. A clear error message instructs users to transfer ownership or delete those workspaces first.
- Disabled (banned) users are explicitly prevented from signing in or re-registering.
- Added new error messages and translations to improve clarity around account deletion restrictions.
- **Bug Fixes**
- Disabled users are now explicitly handled to prevent sign-in attempts.
- **Tests**
- Introduced comprehensive end-to-end tests covering account deletion, banning, and re-registration scenarios.
- **Chores**
- Improved event handling for user deletion and subscription cancellation.
- Updated localization resources with new error messages.
- Renamed payment event handler class for clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-4
fix AI-18
better provider/model choose to allow fallback to similar models (e.g., self-hosted) when the provider is not fully configured
split functions of different output types
fix AI-127
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added automated event handling for workspace updates and document embedding, streamlining document embedding workflows.
- Introduced detection and queuing of documents needing embedding, excluding ignored documents.
- **Improvements**
- Enhanced performance of embedding-related searches by filtering results at the database level.
- Increased concurrency for embedding job processing to improve throughput.
- **Bug Fixes**
- Improved error handling and fallback for missing document titles during embedding.
- Added safeguards to skip invalid embedding jobs based on document identifiers.
- **Tests**
- Expanded test coverage for document embedding and ignored document filtering.
- Updated end-to-end tests to use dynamic content for improved reliability.
- Added synchronization waits in document creation utilities to improve test stability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-207
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced automated periodic indexing of workspaces with a new job type and a scheduled cron job running every 30 seconds.
- Added a unique sequential identifier (`sid`) and an "indexed" flag to workspaces to track indexing status.
- **Improvements**
- Enhanced workspace indexing to handle missing workspaces and snapshots distinctly and selectively index documents.
- Added ability to query workspaces after a given identifier with result limits.
- **Bug Fixes**
- Improved error handling and logging during workspace indexing operations.
- **Tests**
- Expanded test coverage for workspace indexing and auto-indexing, including scheduling and edge cases.
- **Chores**
- Updated data models and schema to support new workspace fields and indexing features.
- Enhanced mock data utilities to allow custom timestamps.
- Improved type safety and flexibility in document snapshot retrieval.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close CLOUD-137
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced advanced workspace-scoped search and aggregation capabilities with support for complex queries, highlights, and pagination.
- Added pluggable search providers: Elasticsearch and Manticoresearch.
- New GraphQL queries, schema types, and resolver support for search and aggregation.
- Enhanced configuration options for search providers in self-hosted and cloud deployments.
- Added Docker Compose services and environment variables for Elasticsearch and Manticoresearch.
- Integrated indexer service into deployment and CI workflows.
- **Bug Fixes**
- Improved error handling with new user-friendly error messages for search provider and indexer issues.
- **Documentation**
- Updated configuration examples and environment variable references for indexer and search providers.
- **Tests**
- Added extensive end-to-end and provider-specific tests covering indexing, searching, aggregation, deletion, and error cases.
- Included snapshot tests and test fixtures for search providers.
- **Chores**
- Updated deployment scripts, Helm charts, and Kubernetes manifests to include indexer-related environment variables and secrets.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-20
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Enhanced file metadata with MIME type, blob ID, and file name across context and workspace, now visible in UI and API.
- Added workspace-level matching for files and documents with configurable thresholds and workspace scoping in search queries.
- Introduced a new error type and user-friendly messaging for global workspace context matching failures.
- **Bug Fixes**
- Improved consistent handling of file MIME types and nullable context IDs for accurate metadata.
- **Documentation**
- Updated GraphQL schema, queries, and mutations to include new metadata fields, optional parameters, and error types.
- **Style**
- Added new localization strings for global context matching error messages.
- **Tests**
- Extended test coverage with new and updated snapshot tests for metadata and matching logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added support for installing self-hosted team licenses via encrypted license files.
- Introduced a new "Onetime" license variant for self-hosted environments.
- Added a GraphQL mutation to upload and install license files.
- License details now display the license variant.
- **Bug Fixes**
- Improved error messages for license activation and expiration, including dynamic reasons.
- **Localization**
- Updated and improved license-related error messages for better clarity.
- **Tests**
- Added comprehensive end-to-end tests for license installation scenarios.
- **Chores**
- Enhanced environment variable handling and public key management for license verification.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Chores**
- Updated a backend dependency to the latest version to ensure continued compatibility and improvements. No user-facing changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sometimes, missing `await` in the test code can cause timing issues, leading to test failures. This PR enables the `no-floating-promises` rule for the test code to ensure that such errors do not occur.
DarkSky