Commit Graph

25 Commits

Author SHA1 Message Date
DarkSky 99332228da feat: native sync state (#14190)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added indexed clock management capabilities for documents, enabling
get, set, and clear operations across Android, iOS, Electron, and web
platforms.

* **Refactor**
* Improved storage architecture to dynamically select platform-specific
implementations (SQLite for Electron, IndexedDB for others).

* **Bug Fixes**
* Enhanced document operations to properly maintain and clean up indexer
synchronization state during document lifecycle changes.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-31 04:09:32 +08:00
DarkSky 2e38898937 feat: refresh index if version changed (#14150) 2025-12-26 01:08:05 +08:00
DarkSky 215541d331 feat: improve indexing perf with native indexer (#14066)
fix #12132, #14006, #13496, #12375, #12132 

The previous idb indexer generated a large number of scattered writes
when flushing to disk, which caused CPU and disk write spikes. If the
document volume is extremely large, the accumulation of write
transactions will cause memory usage to continuously increase.

This PR introduces batch writes to mitigate write performance on the web
side, and adds a native indexer on the Electron side to greatly improve
performance.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Full-text search (FTS) added across storage layers and native plugins:
indexing, search, document retrieval, match ranges, and index flushing.
* New SQLite-backed indexer storage, streaming search/aggregate APIs,
and in-memory index with node-building and highlighting.

* **Performance**
* Indexing rewritten for batched, concurrent writes and parallel
metadata updates.
* Search scoring enhanced to consider multiple term positions and
aggregated term data.

* **Other**
  * Configurable refresh interval and indexer version bump.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-09 22:04:50 +08:00
DarkSky cf4e37c584 feat(native): native reader for indexer (#14055) 2025-12-07 16:22:11 +08:00
Daniel Dybing 88a2e4aa4b fix: improved error description of proxy size limits (#14016)
**Summary:**
This PR improves the user feedback when encountering an HTTP 413
(_CONTENT_TOO_LARGE)_ error caused by a file size limit in the proxy /
ingress controller in a self-hosted environment.

**Example scenario:**
A self-hosted environment serves AFFiNE through an nginx proxy, and the
`client_max_body_size` variable in the configuration file is set to a
smaller size (e.g. 1MB) than AFFiNE's own file size limit (typically
100MB). Previously, the user would get an error saying the file is
larger than 100MB regardless of file size, as all of these cases
resulted in the same internal error. With this fix, the
_CONTENT_TOO_LARGE_ error is now handled separately and gives better
feedback to the user that the failing upload is caused by a fault in the
proxy configuration.

**Screenshot of new error message**

<img width="798" height="171" alt="1MB_now"
src="https://github.com/user-attachments/assets/07b00cd3-ce37-4049-8674-2f3dcb916ab5"
/>


**Affected files:**
  1. packages/common/nbstore/src/storage/errors/over-size.ts
  2. packages/common/nbstore/src/impls/cloud/blob.ts


I'm open to any suggestions in terms of the wording used in the message
to the user. The fix has been tested with an nginx proxy.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved user-facing error messages for file upload failures. When an
upload exceeds the file size limit, users now receive a clearer message
indicating that the upload was stopped by the network proxy due to the
size restriction, providing better understanding of why the upload was
rejected.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-11-27 12:06:07 +08:00
fundon 9702d45c9b fix(nbstore): improve blob size error handling with human-readable limit (#12027)
Closes: [BS-3332](https://linear.app/affine-design/issue/BS-3332/错误信息)

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **Bug Fixes**
	- Improved error messages when file uploads exceed the allowed size, now showing the maximum file size limit in a human-readable format.
	- Enhanced status updates for oversized files by displaying clear error messages during blob uploads.
- **Style**
	- Error messages for file size limits are now more user-friendly and informative.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-07 01:29:37 +00:00
EYHN bc0f32f20b feat(nbstore): add indexer sync version (#11324) 2025-03-31 16:48:47 +00:00
EYHN 8957d0645f feat(nbstore): add indexer storage (#10953) 2025-03-31 12:59:51 +00:00
EYHN 05200ad7b7 feat(nbstore): add blob sync storage (#10752) 2025-03-14 18:05:54 +08:00
EYHN f3ef9c4415 feat(nbstore): rename SyncStorage to DocSyncStorage (#10751) 2025-03-14 06:25:26 +00:00
EYHN a2ffdb4047 feat(core): new worker workspace engine (#9257) 2025-01-16 16:22:18 +00:00
EYHN 46c8c4a408 feat(nbstore): improve nbstore (#9512) 2025-01-06 09:38:04 +00:00
Brooooooklyn 752e697c6a feat(ios): nbstore swift native binding (#9211) 2024-12-27 06:34:30 +00:00
EYHN cbaf35df0b feat(nbstore): add nbstore worker (#9185) 2024-12-20 08:01:23 +00:00
EYHN 64b017dc1b feat(nbstore): remove async on connection api (#9187)
We should not use async on `connect` and `disconnect`, for `WebSocketConnection` will never connect when offline.

We should handle the connection status of each storage in sync, using the `connection.waitForConnect`

This PR also puts the connection reference count on the `connect` and disconnect`
2024-12-18 03:59:49 +00:00
EYHN 3fddf050a4 feat(nbstore): add doc sync state (#9131) 2024-12-18 03:59:47 +00:00
EYHN ffa0231cf5 feat(nbstore): add awareness storage&sync&frontend (#9016) 2024-12-17 04:37:15 +00:00
Brooooooklyn aaaea8918f chore: upgrade to eslint9 (#9163) 2024-12-14 10:29:04 +00:00
Brooooooklyn ea746e3d77 chore: disable rules in oxlint (#9154) 2024-12-13 10:49:35 +00:00
forehalo 8c24f2b906 feat(nbstore): add sqlite implementation (#8811) 2024-12-13 06:13:05 +00:00
EYHN 0c0722c650 feat(nbstore): add blob sync frontend (#9084) 2024-12-11 12:39:26 +00:00
EYHN 35edf389b5 feat(nbstore): better doc sync logic (#9037) 2024-12-10 06:49:21 +00:00
EYHN 7225f59138 feat(nbstore): add blob sync (#8996) 2024-12-07 08:05:04 +00:00
EYHN f54f6e88cb feat(nbstore): new doc sync engine (#8918) 2024-12-07 08:05:02 +00:00
forehalo 4125038ff8 feat(nbstore): init (#7639)
TODO

- [x] basic
- [x] storages
- [x] producer/consumer
- [x] operation pattern
- [x] events
- [x] worker
- [x] readme
- [x] peer dependencies
2024-11-22 03:13:04 +00:00