fix#12132, #14006, #13496, #12375, #12132
The previous idb indexer generated a large number of scattered writes
when flushing to disk, which caused CPU and disk write spikes. If the
document volume is extremely large, the accumulation of write
transactions will cause memory usage to continuously increase.
This PR introduces batch writes to mitigate write performance on the web
side, and adds a native indexer on the Electron side to greatly improve
performance.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Full-text search (FTS) added across storage layers and native plugins:
indexing, search, document retrieval, match ranges, and index flushing.
* New SQLite-backed indexer storage, streaming search/aggregate APIs,
and in-memory index with node-building and highlighting.
* **Performance**
* Indexing rewritten for batched, concurrent writes and parallel
metadata updates.
* Search scoring enhanced to consider multiple term positions and
aggregated term data.
* **Other**
* Configurable refresh interval and indexer version bump.
<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced a Document Summary module, enabling live and cached
document summaries with cloud revalidation.
* Added a feature flag for enabling battery save mode.
* Added explicit pause and resume controls for sync operations,
accessible via UI events and programmatically.
* **Improvements**
* Enhanced sync and indexing logic to support pausing, resuming, and
battery save mode, with improved job prioritization.
* Updated navigation and preview components to use the new document
summary service and improved priority handling.
* Improved logging and state reporting for sync and indexing processes.
* Refined backlink handling with reactive loading states and cloud
revalidation.
* Replaced backlink and link management to use a new dedicated document
links service.
* Enhanced workspace engine to conditionally enable battery save mode
based on feature flags and workspace flavor.
* **Bug Fixes**
* Removed unnecessary debug console logs from various components for
cleaner output.
* **Refactor**
* Replaced battery save mode methods with explicit pause/resume methods
throughout the app and services.
* Modularized and streamlined document summary and sync-related code for
better maintainability.
* Restructured backlink components to improve visibility handling and
data fetching.
* Simplified and improved document backlink data fetching with retry and
loading state management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced a battery save mode that can pause background sync and
processing to conserve energy.
* Battery save mode is automatically enabled when the app window loses
focus and disabled when the window regains focus or is interacted with.
* Available in both web and desktop (Electron) versions.
* **Improvements**
* Added user activity detection to optimize background operations for
better battery efficiency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added enhanced error handling and user-friendly error messages in
quick search and document search menus.
- Introduced loading state indicators for search operations.
- Quick Search now provides explicit error feedback in the UI.
- **Improvements**
- Search and aggregation operations can now prefer remote or local
indexers based on user or system preference.
- Streamlined indexer logic for more consistent and reliable search
experiences.
- Refined error handling in messaging and synchronization layers for
improved stability.
- Enhanced error object handling in messaging for clearer error
propagation.
- Updated cloud workspace storage to always use IndexedDB locally and
CloudIndexer remotely.
- Shifted indexer operations to use synchronized indexer layer for
better consistency.
- Simplified indexer client by consolidating storage and sync layers.
- Improved error propagation in messaging handlers by wrapping error
objects.
- Updated document search to prioritize remote indexer results by
default.
- **Bug Fixes**
- Improved robustness of search features by handling errors gracefully
and preventing potential runtime issues.
- **Style**
- Added new styles for displaying error messages in search interfaces.
- **Chores**
- Removed the obsolete "Enable Cloud Indexer" feature flag; cloud
indexer behavior is now always enabled where applicable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Move the `waitForSynced` method from `frontend` to `nbstore worker` to make the wait more reliable
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added explicit tracking of document updating state to indicate when data is being applied or saved.
- Introduced new methods to wait for update and synchronization completion with abort support.
- **Improvements**
- Applied throttling with leading and trailing emissions to state observables for smoother UI updates.
- Refined synchronization waiting logic for clearer separation between update completion and sync completion.
- Removed throttling in workspace selector component for more immediate state feedback.
- Updated import and clipper services to use the new synchronization waiting methods.
- Simplified asynchronous waiting logic in indexer synchronization methods.
- **Bug Fixes**
- Enhanced accuracy and reliability of document update and sync status indicators.
- **Tests**
- Increased wait timeout in avatar selection test to improve stability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added clear indication and handling of read-only mode for the indexer, including updated status reporting when the indexer is read-only.
- **Bug Fixes**
- Improved state updates to accurately reflect zero activity and completion when in read-only mode.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Closes: [BS-3332](https://linear.app/affine-design/issue/BS-3332/错误信息)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved error messages when file uploads exceed the allowed size, now showing the maximum file size limit in a human-readable format.
- Enhanced status updates for oversized files by displaying clear error messages during blob uploads.
- **Style**
- Error messages for file size limits are now more user-friendly and informative.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Closes: [BS-3137](https://linear.app/affine-design/issue/BS-3137/在-bs-添加-blobstate-接口)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced real-time status tracking for file upload and download operations, allowing users to monitor progress and errors more effectively.
- **Improvements**
- Enhanced icon display for attachments, providing a refreshed visual experience.
- Improved update frequency and accuracy for file synchronization status indicators.
- **Bug Fixes**
- Refined internal logic for filtering status updates, ensuring more precise feedback during file operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
To close [BS-2870](https://linear.app/affine-design/issue/BS-2870/支持识别-和-[-包裹内容为公式)
## Add Markdown Preprocessor Extension and Enhanced LaTeX Support
### Markdown Preprocessor Extension
This PR introduces a new preprocessor extension for Markdown adapters that allows preprocessing of content before conversion:
Adds a new PreprocessorManager for handling text transformations
Introduces extensible preprocessor interface that supports different processing levels (block/slice/doc)
Integrates preprocessor extension into the existing Markdown adapter workflow
### LaTeX Support Enhancement
Extends LaTeX support to handle both traditional and alternative syntax:
Adds support for backslash LaTeX syntax:
Block math: ```\[...\] ``` alongside existing ```$$...$$```
Inline math: ```\(...\) ``` alongside existing ```$...$```
Implements LaTeX preprocessor to standardize syntax before conversion
Updates tests to cover both syntax variants
DarkSky