mirror of https://github.com/toeverything/AFFiNE.git synced 2026-02-13 21:05:19 +00:00

Files

DarkSky 215541d331 feat: improve indexing perf with native indexer (#14066 )

fix #12132, #14006, #13496, #12375, #12132 

The previous idb indexer generated a large number of scattered writes
when flushing to disk, which caused CPU and disk write spikes. If the
document volume is extremely large, the accumulation of write
transactions will cause memory usage to continuously increase.

This PR introduces batch writes to mitigate write performance on the web
side, and adds a native indexer on the Electron side to greatly improve
performance.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Full-text search (FTS) added across storage layers and native plugins:
indexing, search, document retrieval, match ranges, and index flushing.
* New SQLite-backed indexer storage, streaming search/aggregate APIs,
and in-memory index with node-building and highlighting.

* **Performance**
* Indexing rewritten for batched, concurrent writes and parallel
metadata updates.
* Search scoring enhanced to consider multiple term positions and
aggregated term data.

* **Other**
  * Configurable refresh interval and indexer version bump.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

2025-12-09 22:04:50 +08:00

src

feat: improve indexing perf with native indexer (#14066 )

2025-12-09 22:04:50 +08:00

package.json

chore: bump deps (#14065 )

2025-12-08 21:47:25 +08:00

README.md

feat(nbstore): init (#7639 )

2024-11-22 03:13:04 +00:00

tsconfig.json

feat(common): add blocksuite reader lib (#11955 )

2025-05-14 14:52:41 +00:00

README.md

Space Storage

Usage

Independent Storage usage

import type { ConnectionStatus } from '@affine/nbstore';
import { IndexedDBDocStorage } from '@affine/nbstore/idb';

const storage = new IndexedDBDocStorage({
  peer: 'local'
  spaceId: 'my-new-workspace',
});

await storage.connect();
storage.connection.onStatusChange((status: ConnectionStatus, error?: Error) => {
  ui.show(status, error);
});

// { docId: string, bin: Uint8Array, timestamp: Date, editor?: string } | null
const doc = await storage.getDoc('my-first-doc');

Use All storages together

import { SpaceStorage } from '@affine/nbstore';
import type { ConnectionStatus } from '@affine/nbstore';
import { IndexedDBDocStorage } from '@affine/nbstore/idb';
import { SqliteBlobStorage } from '@affine/nbstore/sqlite';

const storage = new SpaceStorage([new IndexedDBDocStorage({}), new SqliteBlobStorage({})]);

await storage.connect();
storage.on('connection', ({ storage, status, error }) => {
  ui.show(storage, status, error);
});

await storage.get('doc').pushDocUpdate({ docId: 'my-first-doc', bin: new Uint8Array(), editor: 'me' });
await storage.tryGet('blob')?.get('img');

Put Storage behind Worker

import { SpaceStorageWorkerClient } from '@affine/nbstore/op';
import type { ConnectionStatus } from '@affine/nbstore';
import { IndexedDBDocStorage } from '@affine/nbstore/idb';

const client = new SpaceStorageWorkerClient();
client.addStorage(IndexedDBDocStorage, {
  // options can only be structure-cloneable type
  peer: 'local',
  spaceType: 'workspace',
  spaceId: 'my-new-workspace',
});

await client.connect();
client.ob$('connection', ({ storage, status, error }) => {
  ui.show(storage, status, error);
});

await client.call('pushDocUpdate', { docId: 'my-first-doc', bin: new Uint8Array(), editor: 'me' });

// call unregistered op will leads to Error
// Error { message: 'Handler for operation [listHistory] is not registered.' }
await client.call('listHistories', { docId: 'my-first-doc' });