Commit Graph

96 Commits

Author SHA1 Message Date
DarkSky e7d0f31546 fix: blob redirect 2026-01-04 00:23:51 +08:00
DarkSky 99332228da feat: native sync state (#14190)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added indexed clock management capabilities for documents, enabling
get, set, and clear operations across Android, iOS, Electron, and web
platforms.

* **Refactor**
* Improved storage architecture to dynamically select platform-specific
implementations (SQLite for Electron, IndexedDB for others).

* **Bug Fixes**
* Enhanced document operations to properly maintain and clean up indexer
synchronization state during document lifecycle changes.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-31 04:09:32 +08:00
DarkSky 6514614df8 feat: bump electron (#14158) 2025-12-27 23:54:11 +08:00
DarkSky 78949044ec feat: improve idb perf (#14159)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Performance**
* Optimized database operations through improved batch processing to
accelerate data retrieval, updates, and deletion operations for better
efficiency.

* **Reliability**
* Enhanced transaction durability handling to strengthen data
consistency and ensure more reliable persistence of database changes and
updates.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-27 08:22:37 +08:00
DarkSky 2e38898937 feat: refresh index if version changed (#14150) 2025-12-26 01:08:05 +08:00
DarkSky e8693a3a25 feat: introduce fuzzy search for native indexer (#14109) 2025-12-25 04:40:23 +08:00
DarkSky 76524084d1 feat: multipart blob sync support (#14138)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Flexible blob uploads: GRAPHQL, presigned, and multipart flows with
per‑part URLs, abort/complete operations, presigned proxy endpoints, and
nightly cleanup of expired pending uploads.

* **API / Schema**
* GraphQL additions: new types, mutations, enum and error to manage
upload lifecycle (create, complete, abort, get part URL).

* **Database**
* New blob status enum and columns (status, upload_id); listing now
defaults to completed blobs.

* **Localization**
  * Added user-facing message: "Blob is invalid."

* **Tests**
* Expanded unit and end‑to‑end coverage for upload flows, proxy
behavior, multipart and provider integrations.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-23 22:09:21 +08:00
DarkSky cb0ff04efa feat: bump more deps (#14079) 2025-12-10 16:02:28 +08:00
DarkSky 215541d331 feat: improve indexing perf with native indexer (#14066)
fix #12132, #14006, #13496, #12375, #12132 

The previous idb indexer generated a large number of scattered writes
when flushing to disk, which caused CPU and disk write spikes. If the
document volume is extremely large, the accumulation of write
transactions will cause memory usage to continuously increase.

This PR introduces batch writes to mitigate write performance on the web
side, and adds a native indexer on the Electron side to greatly improve
performance.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Full-text search (FTS) added across storage layers and native plugins:
indexing, search, document retrieval, match ranges, and index flushing.
* New SQLite-backed indexer storage, streaming search/aggregate APIs,
and in-memory index with node-building and highlighting.

* **Performance**
* Indexing rewritten for batched, concurrent writes and parallel
metadata updates.
* Search scoring enhanced to consider multiple term positions and
aggregated term data.

* **Other**
  * Configurable refresh interval and indexer version bump.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-09 22:04:50 +08:00
DarkSky cf14accd2b fix: unstable test 2025-12-07 20:22:43 +08:00
DarkSky cf4e37c584 feat(native): native reader for indexer (#14055) 2025-12-07 16:22:11 +08:00
Daniel Dybing 88a2e4aa4b fix: improved error description of proxy size limits (#14016)
**Summary:**
This PR improves the user feedback when encountering an HTTP 413
(_CONTENT_TOO_LARGE)_ error caused by a file size limit in the proxy /
ingress controller in a self-hosted environment.

**Example scenario:**
A self-hosted environment serves AFFiNE through an nginx proxy, and the
`client_max_body_size` variable in the configuration file is set to a
smaller size (e.g. 1MB) than AFFiNE's own file size limit (typically
100MB). Previously, the user would get an error saying the file is
larger than 100MB regardless of file size, as all of these cases
resulted in the same internal error. With this fix, the
_CONTENT_TOO_LARGE_ error is now handled separately and gives better
feedback to the user that the failing upload is caused by a fault in the
proxy configuration.

**Screenshot of new error message**

<img width="798" height="171" alt="1MB_now"
src="https://github.com/user-attachments/assets/07b00cd3-ce37-4049-8674-2f3dcb916ab5"
/>


**Affected files:**
  1. packages/common/nbstore/src/storage/errors/over-size.ts
  2. packages/common/nbstore/src/impls/cloud/blob.ts


I'm open to any suggestions in terms of the wording used in the message
to the user. The fix has been tested with an nginx proxy.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved user-facing error messages for file upload failures. When an
upload exceeds the file size limit, users now receive a clearer message
indicating that the upload was stopped by the network proxy due to the
size restriction, providing better understanding of why the upload was
rejected.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-11-27 12:06:07 +08:00
EYHN 1ceed6c145 feat(core): support better battery save mode (#13383)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Introduced a Document Summary module, enabling live and cached
document summaries with cloud revalidation.
  * Added a feature flag for enabling battery save mode.
* Added explicit pause and resume controls for sync operations,
accessible via UI events and programmatically.

* **Improvements**
* Enhanced sync and indexing logic to support pausing, resuming, and
battery save mode, with improved job prioritization.
* Updated navigation and preview components to use the new document
summary service and improved priority handling.
* Improved logging and state reporting for sync and indexing processes.
* Refined backlink handling with reactive loading states and cloud
revalidation.
* Replaced backlink and link management to use a new dedicated document
links service.
* Enhanced workspace engine to conditionally enable battery save mode
based on feature flags and workspace flavor.

* **Bug Fixes**
* Removed unnecessary debug console logs from various components for
cleaner output.

* **Refactor**
* Replaced battery save mode methods with explicit pause/resume methods
throughout the app and services.
* Modularized and streamlined document summary and sync-related code for
better maintainability.
* Restructured backlink components to improve visibility handling and
data fetching.
* Simplified and improved document backlink data fetching with retry and
loading state management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-08-01 08:31:31 +00:00
EYHN b2bf591463 feat(core): bettery save mode (#12996)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Introduced a battery save mode that can pause background sync and
processing to conserve energy.
* Battery save mode is automatically enabled when the app window loses
focus and disabled when the window regains focus or is interacted with.
  * Available in both web and desktop (Electron) versions.

* **Improvements**
* Added user activity detection to optimize background operations for
better battery efficiency.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-07-03 03:48:17 +00:00
EYHN aa4874a55c feat(core): use cloud indexer for search (#12899)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added enhanced error handling and user-friendly error messages in
quick search and document search menus.
  - Introduced loading state indicators for search operations.
  - Quick Search now provides explicit error feedback in the UI.

- **Improvements**
- Search and aggregation operations can now prefer remote or local
indexers based on user or system preference.
- Streamlined indexer logic for more consistent and reliable search
experiences.
- Refined error handling in messaging and synchronization layers for
improved stability.
- Enhanced error object handling in messaging for clearer error
propagation.
- Updated cloud workspace storage to always use IndexedDB locally and
CloudIndexer remotely.
- Shifted indexer operations to use synchronized indexer layer for
better consistency.
  - Simplified indexer client by consolidating storage and sync layers.
- Improved error propagation in messaging handlers by wrapping error
objects.
- Updated document search to prioritize remote indexer results by
default.

- **Bug Fixes**
- Improved robustness of search features by handling errors gracefully
and preventing potential runtime issues.

- **Style**
  - Added new styles for displaying error messages in search interfaces.

- **Chores**
- Removed the obsolete "Enable Cloud Indexer" feature flag; cloud
indexer behavior is now always enabled where applicable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-06-25 02:55:27 +00:00
EYHN b847de4980 fix(core): remove quota modal (#12586)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **Removed Features**
  - Removed all quota-reached modal dialogs, including both cloud and local storage quota notifications.
  - Users will no longer see modal alerts when storage limits are reached in workspaces.
- **User Interface**
  - Quota-reached modals and related styles have been removed from workspace layouts on both desktop and mobile.
- **Other Changes**
  - Quota notification logic and related settings have been eliminated from the application.
  - Maximum blob size enforcement and related callbacks have been removed from blob management.
  - Localization entries related to file upload size limits and quota tips have been removed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-28 08:04:22 +00:00
EYHN 9c5af576ee feat(nbstore): add more blob sync state (#12516) 2025-05-27 02:20:49 +00:00
EYHN 4aa9ae5e68 fix(nbstore): fix http request timeout handling (#12515)
`AbortSignal.timeout` will cause a timeout when transferring large blobs.  by using `setTimeout` currently, the timeout only covers the phase of establishing a connection.

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **Bug Fixes**
  - Improved reliability of request cancellation and timeout handling for HTTP requests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-27 02:20:49 +00:00
fundon 952650e1bc refactor(nbstore): improve HTTP connection timeout handling (#11985)
Closes: [BS-3344](https://linear.app/affine-design/issue/BS-3344/改进请求超时提示)

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **Refactor**
  - Improved internal request timeout handling for cloud operations, resulting in more robust and reliable network requests. No changes to user-facing features or functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-23 09:13:59 +00:00
EYHN 59ef4b227b refactor(nbstore): improve doc state management (#12359)
Move the `waitForSynced` method from `frontend` to `nbstore worker` to make the wait more reliable

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **New Features**
  - Added explicit tracking of document updating state to indicate when data is being applied or saved.
  - Introduced new methods to wait for update and synchronization completion with abort support.

- **Improvements**
  - Applied throttling with leading and trailing emissions to state observables for smoother UI updates.
  - Refined synchronization waiting logic for clearer separation between update completion and sync completion.
  - Removed throttling in workspace selector component for more immediate state feedback.
  - Updated import and clipper services to use the new synchronization waiting methods.
  - Simplified asynchronous waiting logic in indexer synchronization methods.

- **Bug Fixes**
  - Enhanced accuracy and reliability of document update and sync status indicators.

- **Tests**
  - Increased wait timeout in avatar selection test to improve stability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-20 07:33:20 +00:00
EYHN 3c982d2b91 fix(nbstore): fix readonly mode indexer status (#12353)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **New Features**
  - Added clear indication and handling of read-only mode for the indexer, including updated status reporting when the indexer is read-only.
- **Bug Fixes**
  - Improved state updates to accurately reflect zero activity and completion when in read-only mode.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-20 04:01:04 +00:00
fengmk2 4e37a1322e feat(nbstore): add cloud indexer storage (#12245)
close AF-2613
2025-05-19 04:39:39 +00:00
fengmk2 6792c3e656 feat(common): add blocksuite reader lib (#11955)
close CLOUD-202
2025-05-14 14:52:41 +00:00
EYHN 52253e9e82 fix(nbstore): update workspace blob quota query (#12191)
Change the query for querying quota of Cloud Blob Storage.

The original query used new fields, which caused errors in the old version of the server. This PR uses a simpler query to ensure compatibility.

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **New Features**
  - Added the ability to retrieve a workspace's blob storage quota, including both the raw limit and a human-readable format, via a new query.

- **Bug Fixes**
  - Updated quota retrieval to use the new blob-specific quota query for improved accuracy.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-08 09:14:54 +00:00
fundon af7cbdaf7e fix(nbstore): should not force throw error (#12152)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **Bug Fixes**
  - Improved handling of oversized blobs by allowing operations to continue after notifying about size limits, instead of stopping execution with an error.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-07 02:54:26 +00:00
fundon 9702d45c9b fix(nbstore): improve blob size error handling with human-readable limit (#12027)
Closes: [BS-3332](https://linear.app/affine-design/issue/BS-3332/错误信息)

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **Bug Fixes**
	- Improved error messages when file uploads exceed the allowed size, now showing the maximum file size limit in a human-readable format.
	- Enhanced status updates for oversized files by displaying clear error messages during blob uploads.
- **Style**
	- Error messages for file size limits are now more user-friendly and informative.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-07 01:29:37 +00:00
liuyi 0665d20d67 fix(nbstore): wrong order of socketio transports (#12077) 2025-04-30 03:05:40 +00:00
EYHN a75ae38308 fix(nbstore): fix cloud blob in ios and android (#11938) 2025-04-30 10:25:15 +08:00
EYHN 08d1b8dcbf fix(core): better search result (#12015)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **Refactor**
	- Simplified search menu logic by removing unnecessary filtering and parameters from menu item generation. No changes to visible functionality.
- **Bug Fixes**
	- Improved search index matching to retrieve all relevant entries, enhancing search accuracy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-04-28 03:55:45 +00:00
fundon e9003dec9e feat(editor): add blobState$ to BlobEngine (#11756)
Closes: [BS-3137](https://linear.app/affine-design/issue/BS-3137/在-bs-添加-blobstate-接口)

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **New Features**
  - Introduced real-time status tracking for file upload and download operations, allowing users to monitor progress and errors more effectively.
- **Improvements**
  - Enhanced icon display for attachments, providing a refreshed visual experience.
  - Improved update frequency and accuracy for file synchronization status indicators.
- **Bug Fixes**
  - Refined internal logic for filtering status updates, ensuring more precise feedback during file operations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-04-28 10:46:46 +08:00
EYHN 4df75ec5d0 fix(core): subscribe search not unsubscribe (#11929) 2025-04-23 13:56:58 +00:00
Saul-Mirone 16831e8c0e feat(editor): reference, mention and latex inline extension (#11914)
Closes: BS-3215
Closes: BS-3218
Closes: BS-3217
2025-04-23 09:22:16 +00:00
EYHN 9fbffccc25 fix(nbstore): fix indexer cache not working (#11922) 2025-04-23 08:53:48 +00:00
EYHN 5694e3a56c feat(nbstore): add internal timeout to autoreconnection (#11785) 2025-04-18 04:20:52 +00:00
EYHN dd51180acb feat(core): replace onboarding template (#11768) 2025-04-18 02:31:13 +00:00
EYHN 5e9ad634b7 feat(nbstore): optimize search performance (#11778)
now we can debug indexeddb performance by devtool

![image.png](https://graphite-user-uploaded-assets-prod.s3.amazonaws.com/g3jz87HxbjOJpXV3FPT7/75cf686f-9bc4-4db3-86ac-12bba83e8d83.png)
2025-04-17 13:16:18 +00:00
EYHN a759a1988e fix(nbstore): connect before do operation (#11569) 2025-04-10 16:05:46 +08:00
liuyi e376aa57c5 chore: bring treeshake back (#11577) 2025-04-10 14:12:36 +08:00
EYHN 1bd31b67cd feat(core): improve priority queue performance (#11559) 2025-04-09 14:56:32 +00:00
donteatfriedrice 568a390b75 feat(editor): support markdown adapter preprocessed with latex delimiters (#11431)
To close [BS-2870](https://linear.app/affine-design/issue/BS-2870/支持识别-和-[-包裹内容为公式)

## Add Markdown Preprocessor Extension and Enhanced LaTeX Support

### Markdown Preprocessor Extension
This PR introduces a new preprocessor extension for Markdown adapters that allows preprocessing of content before conversion:

Adds a new PreprocessorManager for handling text transformations
Introduces extensible preprocessor interface that supports different processing levels (block/slice/doc)

Integrates preprocessor extension into the existing Markdown adapter workflow

### LaTeX Support Enhancement
Extends LaTeX support to handle both traditional and alternative syntax:
Adds support for backslash LaTeX syntax:

Block math: ```\[...\] ``` alongside existing ```$$...$$```
Inline math: ```\(...\) ``` alongside existing ```$...$```

Implements LaTeX preprocessor to standardize syntax before conversion

Updates tests to cover both syntax variants
2025-04-07 02:18:04 +00:00
EYHN 521eb09a7a fix(nbstore): adjust indexer logic (#11355) 2025-04-01 11:31:58 +00:00
EYHN bc0f32f20b feat(nbstore): add indexer sync version (#11324) 2025-03-31 16:48:47 +00:00
EYHN 8957d0645f feat(nbstore): add indexer storage (#10953) 2025-03-31 12:59:51 +00:00
EYHN 7091111f85 feat(nbstore): add upload function to blob frontend (#11247) 2025-03-27 12:11:30 +00:00
EYHN dda3103d1b feat(nbstore): allow polling protocol (#11160) 2025-03-25 08:15:23 +00:00
EYHN edcf6a967a feat(nbstore): blob storage adapt to new error type (#11005) 2025-03-21 16:05:30 +08:00
EYHN 7dbc9b42b7 feat(nbstore): add debug log for incorrect writing (#10697) 2025-03-17 09:09:27 +00:00
EYHN 5915e6a6f1 fix(nbstore): fix download blob return too early (#10875) 2025-03-14 16:41:31 +00:00
EYHN 05200ad7b7 feat(nbstore): add blob sync storage (#10752) 2025-03-14 18:05:54 +08:00
EYHN f3ef9c4415 feat(nbstore): rename SyncStorage to DocSyncStorage (#10751) 2025-03-14 06:25:26 +00:00