Commit Graph

808 Commits

Author SHA1 Message Date
DarkSky f373e08583 feat: refactor doc write in native (#14272) 2026-01-18 16:31:12 +08:00
realies 0da91e406e feat(server): add document write tools for mcp (#14245)
## Summary

This PR adds write capabilities to AFFiNE's MCP (Model Context Protocol)
integration, enabling external tools (Claude, GPT, etc.) to create and
modify documents programmatically.

**New MCP Tools:**
- `create_document` - Create new documents from markdown content
- `update_document` - Update document content using structural diffing
for minimal changes (preserves document history and enables real-time
collaboration)

**Implementation:**
- `markdown_to_ydoc.rs` - Converts markdown to AFFiNE-compatible y-octo
binary format
- `markdown_utils.rs` - Shared markdown parsing utilities (used by both
ydoc-to-md and md-to-ydoc)
- `update_ydoc.rs` - Structural diffing implementation for updating
existing documents
- `DocWriter` service - TypeScript service for document operations
- Exposes `markdownToDocBinary` and `updateDocBinary` via napi bindings

**Supported Markdown Elements:**
- Headings (H1-H6)
- Paragraphs
- Bullet lists and numbered lists
- Code blocks (with language detection)
- Blockquotes
- Horizontal dividers
- Todo items (checkboxes)

**y-octo Changes:**
This PR reverts the y-octo sync (ca2462f, a5b60cf) which introduced a
concurrency bug causing hangs when creating documents with many nested
block structures. It also ports the improved `get_node_index` binary
search fix from upstream that prevents divide-by-zero panics when
decoding documents.

## Test Results 

### Unit Tests (47/47 passing)

| Test Suite | Tests | Status |
|------------|-------|--------|
| markdown_to_ydoc | 16/16 |  Pass |
| markdown_utils | 11/11 |  Pass |
| update_ydoc | 13/13 |  Pass |
| delta_markdown | 2/2 |  Pass |
| affine (doc parser) | 5/5 |  Pass |

### End-to-End MCP Testing 

Tested against local AFFiNE server with real MCP client requests:

| Tool | Result | Notes |
|------|--------|-------|
| `tools/list` |  Pass | Returns all 5 tools with correct schemas |
| `create_document` |  Pass | Successfully created test documents |
| `update_document` |  Pass | Successfully updated documents with
structural diffing |
| `read_document` |  Pass | Existing tool, works correctly |
| `keyword_search` |  Pass | Existing tool, works correctly |

**E2E Test Details:**
- Started local AFFiNE server with PostgreSQL, Redis, and Manticore
- Created test user and workspace via seed/GraphQL
- Verified MCP endpoint at `/api/workspaces/:workspaceId/mcp`
- Tested JSON-RPC calls with proper SSE streaming
- Confirmed documents are stored and indexed correctly (verified via
server logs)

## Test Plan
- [x] All Rust unit tests pass (47 tests)
- [x] Native bindings build successfully (release mode)
- [x] Document creation via MCP works end-to-end
- [x] Document update via MCP works end-to-end
- [x] CodeRabbit feedback addressed
- [ ] Integration testing with Claude/GPT MCP clients

Closes #14161

---

**Requested by:** @realies  
**Key guidance from:** @darkskygit (use y-octo instead of yjs for memory
efficiency)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Create documents from Markdown: generate new documents directly from
Markdown content with automatic title extraction
* Update documents with Markdown: modify existing documents using
Markdown as the source with automatic diff calculation for efficient
updates
* Copilot integration: new tools for document creation and updates
through Copilot's interface

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-16 20:57:24 +08:00
DarkSky 2c5559ed0b fix: missing session fields 2026-01-16 17:26:05 +08:00
DarkSky 924d58603f chore: improve event flow (#14266) 2026-01-16 16:07:27 +08:00
DarkSky ac7a95e708 feat: misc optimizations (#14257)
fix #13798
2026-01-15 03:47:22 +08:00
DarkSky fc59dff9e2 chore: enable blur background for mac 2026-01-15 00:35:11 +08:00
DarkSky 27a58e764c chore: bump version & deps 2026-01-15 00:33:51 +08:00
DarkSky 7c24b2521a feat: reduce backend (#14251)
#### PR Dependency Tree


* **PR #14251** 👈

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Current user profile now exposes access tokens, revealed tokens, and
detailed calendar accounts/subscriptions.
* Workspace now exposes permissions, calendars, calendar events, and a
workspace-scoped blob upload part URL.
  * New document-update mutation for applying doc updates.

* **API Changes**
  * validateAppConfig is now a query (mutation deprecated).
* Several legacy top-level calendar/blob endpoints deprecated in favor
of user/workspace fields.

* **Refactor**
* Calendar, blob-upload and access-token surfaces reorganized to use
user/workspace-centric fields.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-14 00:01:07 +08:00
DarkSky 279b7bb64f feat(core): integrate google calendar sync (#14248)
fix #14170 
fix #13893 
fix #13673 
fix #13543 
fix #13308 
fix #7607




#### PR Dependency Tree


* **PR #14247**
  * **PR #14248** 👈

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Integrations panel in Account Settings to link/unlink calendar
providers.
  * Collapsible settings wrapper for improved layout.

* **Improvements**
* Calendar system reworked: per-account calendar groups, simplified
toggles with explicit Save, richer event display (multi-dot date
indicators), improved event time/title handling across journal views.

* **Localization**
* Added calendar keys: save-error, no-journal, no-calendar; removed
legacy duplicate-error keys.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-13 02:38:16 +08:00
DarkSky 89f0430242 fix: race conditions (y-crdt/y-octo#51) 2026-01-13 01:10:07 +08:00
DarkSky 0bd8160ed4 feat: init cloud calendar support (#14247)
#### PR Dependency Tree


* **PR #14247** 👈
  * **PR #14248**

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Google Calendar integration (disabled by default): link/unlink
accounts, OAuth flow, webhooks, real-time push, background sync,
workspace calendars with customizable items and date-range event
viewing.
* **GraphQL / Client**
* New queries & mutations for accounts, subscriptions, events,
providers, and workspace calendar management.
* **Localization**
* Added localized error message for calendar provider request failures.
* **Tests**
* Backend tests covering sync, webhook renewal, and error/error-recovery
scenarios.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-12 23:17:43 +08:00
DarkSky a5b60cf679 fix: index calc & detached node handle (y-crdt/y-octo#50) 2026-01-11 18:44:55 +08:00
DarkSky ca2462f987 feat(native): sync yocto codes (#14243)
#### PR Dependency Tree


* **PR #14243** 👈

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Batch management API for coordinated document mutations and change
tracking.
* New document accessors (IDs, state snapshots, change/delete set
queries) and subscriber count.

* **Chores**
  * Upgraded Rust edition across packages to 2024.
  * Repository-wide formatting, stylistic cleanups and test adjustments.

* **Breaking Changes**
* Removed the Node native bindings package and its JS/TS declarations
and tests (no longer published/available).

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-11 06:08:33 +08:00
DarkSky e7d0f31546 fix: blob redirect 2026-01-04 00:23:51 +08:00
DarkSky fe5d6c0c0f feat(editor): support frontmatter & colored text parsing (#14205)
fix #13847
2026-01-03 22:43:11 +08:00
DarkSky 9a7f8e7d4d feat: workspace level share settings (#14201)
fix #13698
2026-01-03 01:13:27 +08:00
DarkSky 60de882a30 feat: shared link list (#14200)
#### PR Dependency Tree


* **PR #14200** 👈

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **New Features**
* Added a "Shared Links" panel to workspace management, enabling admins
to view all published documents within a workspace
* Added publication date tracking for published documents, now displayed
alongside shared links

* **Chores**
  * Removed deprecated `publicPages` field; use `publicDocs` instead

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-02 21:07:41 +08:00
DarkSky f745f7b669 feat: pre-aggregation workspace stats 2026-01-01 05:01:52 +08:00
DarkSky 99332228da feat: native sync state (#14190)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added indexed clock management capabilities for documents, enabling
get, set, and clear operations across Android, iOS, Electron, and web
platforms.

* **Refactor**
* Improved storage architecture to dynamically select platform-specific
implementations (SQLite for Electron, IndexedDB for others).

* **Bug Fixes**
* Enhanced document operations to properly maintain and clean up indexer
synchronization state during document lifecycle changes.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-31 04:09:32 +08:00
DarkSky e2adab7805 fix: table compatibility (#14185) 2025-12-30 12:01:27 +08:00
DarkSky ff2e96d847 feat: database indexing support (#14181) 2025-12-30 05:23:09 +08:00
DarkSky 95a5e941e7 feat: improve admin panel (#14180) 2025-12-30 05:22:54 +08:00
Yii 1b9d065778 chore(yocto): should auto gc after applying updates (#12199)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Improved document update handling to optimize storage automatically
when garbage collection is enabled.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: DarkSky <25152247+darkskygit@users.noreply.github.com>
2025-12-29 21:48:23 +08:00
DarkSky 6514614df8 feat: bump electron (#14158) 2025-12-27 23:54:11 +08:00
DarkSky 78949044ec feat: improve idb perf (#14159)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Performance**
* Optimized database operations through improved batch processing to
accelerate data retrieval, updates, and deletion operations for better
efficiency.

* **Reliability**
* Enhanced transaction durability handling to strengthen data
consistency and ensure more reliable persistence of database changes and
updates.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-27 08:22:37 +08:00
DarkSky 4eed92cebf feat: improve electron sandbox (#14156) 2025-12-27 03:23:28 +08:00
DarkSky 2e38898937 feat: refresh index if version changed (#14150) 2025-12-26 01:08:05 +08:00
DarkSky e8693a3a25 feat: introduce fuzzy search for native indexer (#14109) 2025-12-25 04:40:23 +08:00
DarkSky 4b721dffe0 feat: set admin name when self hosted init (#14146)
fix #14134
2025-12-23 23:38:34 +08:00
DarkSky 76524084d1 feat: multipart blob sync support (#14138)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Flexible blob uploads: GRAPHQL, presigned, and multipart flows with
per‑part URLs, abort/complete operations, presigned proxy endpoints, and
nightly cleanup of expired pending uploads.

* **API / Schema**
* GraphQL additions: new types, mutations, enum and error to manage
upload lifecycle (create, complete, abort, get part URL).

* **Database**
* New blob status enum and columns (status, upload_id); listing now
defaults to completed blobs.

* **Localization**
  * Added user-facing message: "Blob is invalid."

* **Tests**
* Expanded unit and end‑to‑end coverage for upload flows, proxy
behavior, multipart and provider integrations.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-23 22:09:21 +08:00
DarkSky 321965a424 feat: impl text delta support (#14132) 2025-12-22 03:16:16 +08:00
DarkSky 844b9d9592 feat(server): impl native reader for server (#14100) 2025-12-14 00:28:43 +08:00
DarkSky cb0ff04efa feat: bump more deps (#14079) 2025-12-10 16:02:28 +08:00
DarkSky 40f3337d45 feat: bump deps (#14076)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Updated core dependencies, developer tooling and Rust toolchain to
newer stable versions across the repo
* Upgraded Storybook to v10 and improved ESM path resolution for
storybook tooling
* Broadened native binding platform/architecture support and
strengthened native module version validation, loading and WASI handling

* **New Features**
* Exposed an additional native text export for consumers (enhanced
JS/native surface)

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-10 03:52:14 +08:00
DarkSky 215541d331 feat: improve indexing perf with native indexer (#14066)
fix #12132, #14006, #13496, #12375, #12132 

The previous idb indexer generated a large number of scattered writes
when flushing to disk, which caused CPU and disk write spikes. If the
document volume is extremely large, the accumulation of write
transactions will cause memory usage to continuously increase.

This PR introduces batch writes to mitigate write performance on the web
side, and adds a native indexer on the Electron side to greatly improve
performance.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Full-text search (FTS) added across storage layers and native plugins:
indexing, search, document retrieval, match ranges, and index flushing.
* New SQLite-backed indexer storage, streaming search/aggregate APIs,
and in-memory index with node-building and highlighting.

* **Performance**
* Indexing rewritten for batched, concurrent writes and parallel
metadata updates.
* Search scoring enhanced to consider multiple term positions and
aggregated term data.

* **Other**
  * Configurable refresh interval and indexer version bump.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-09 22:04:50 +08:00
DarkSky 027f741ed6 chore: bump deps (#14065)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Updated dependency versions across the monorepo (notably zod →
^3.25.76 and vitest-related packages → ^3.2.4), plus minor package bumps
to align tooling and libraries. These are manifest/test-tooling updates
only; no public API, behavior, or end-user features were changed.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-08 21:47:25 +08:00
DarkSky 776ca2c702 chore: bump version 2025-12-08 10:47:37 +08:00
DarkSky cf14accd2b fix: unstable test 2025-12-07 20:22:43 +08:00
DarkSky cf4e37c584 feat(native): native reader for indexer (#14055) 2025-12-07 16:22:11 +08:00
Daniel Dybing 88a2e4aa4b fix: improved error description of proxy size limits (#14016)
**Summary:**
This PR improves the user feedback when encountering an HTTP 413
(_CONTENT_TOO_LARGE)_ error caused by a file size limit in the proxy /
ingress controller in a self-hosted environment.

**Example scenario:**
A self-hosted environment serves AFFiNE through an nginx proxy, and the
`client_max_body_size` variable in the configuration file is set to a
smaller size (e.g. 1MB) than AFFiNE's own file size limit (typically
100MB). Previously, the user would get an error saying the file is
larger than 100MB regardless of file size, as all of these cases
resulted in the same internal error. With this fix, the
_CONTENT_TOO_LARGE_ error is now handled separately and gives better
feedback to the user that the failing upload is caused by a fault in the
proxy configuration.

**Screenshot of new error message**

<img width="798" height="171" alt="1MB_now"
src="https://github.com/user-attachments/assets/07b00cd3-ce37-4049-8674-2f3dcb916ab5"
/>


**Affected files:**
  1. packages/common/nbstore/src/storage/errors/over-size.ts
  2. packages/common/nbstore/src/impls/cloud/blob.ts


I'm open to any suggestions in terms of the wording used in the message
to the user. The fix has been tested with an nginx proxy.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved user-facing error messages for file upload failures. When an
upload exceeds the file size limit, users now receive a clearer message
indicating that the upload was stopped by the network proxy due to the
size restriction, providing better understanding of why the upload was
rejected.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-11-27 12:06:07 +08:00
DarkSky d90eeffe84 fix: ci 2025-11-15 23:52:14 +08:00
DarkSky 29cb937493 feat(native): add events feature for yocto 2025-11-15 21:17:02 +08:00
DarkSky 4e082e4170 chore: bump version 2025-11-15 17:29:54 +08:00
DarkSky 7e6ead4232 feat: native doc reader (#13881) 2025-11-08 23:07:16 +08:00
Hongxu Xu 02dcfdcc40 chore(native): remove unused deps (#13854)
Identified some unused dependencies and refactored them.
2025-11-07 11:31:19 +08:00
DarkSky 89cc9b072b feat(server): early subscription for iap (#13826)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a mutation to request/apply a subscription by transaction ID
(client mutation and server operation), returning subscription details.
* **Bug Fixes / Improvements**
* More robust external subscription sync with safer conflict detection,
optional short-lived confirmation, improved parsing and error logging.
* **Chores**
  * Standardized time constants for clarity.
* **Tests**
* Updated subscription test data (expiration date) to reflect new
lifecycle expectations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-10-29 14:25:43 +00:00
DarkSky e4b5b24fdd chore: bump package version 2025-10-29 21:14:34 +08:00
Juan Abimael Santos Castillo 6fb0ff9177 chore(native): simplify Array::get method with functional approach (#13771)
## Issue

The Array::get method used verbose nested Option handling with redundant
returns, making the code longer and less Rust idiomatic.

## Solution

Replaced the nested if let structure with a functional and_then approach
that:
- Eliminates redundant return keyword usage
- Flattens Option handling for cleaner logic
- Reduces code from 13 lines to 9 lines
- Maintains identical functionality

## Result

More idiomatic Rust code that's easier to read and maintain, with all
tests passing.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Refactor**
* Improved internal code structure with functional programming patterns
for enhanced maintainability. No visible changes to end-user
functionality.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: DarkSky <25152247+darkskygit@users.noreply.github.com>
2025-10-21 16:45:58 +00:00
DarkSky 2d1caff45c feat(server): refresh subscription (#13670)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an on-demand mutation to refresh the current user's
subscriptions, syncing with RevenueCat when applicable and handling
Stripe-only cases.
* Subscription variant normalization for clearer plan information and
consistent results.

* **Tests**
* Added tests for refresh behavior: empty state, RevenueCat-backed
multi-step sync, and Stripe-only scenarios.

* **Client**
* New client operation to invoke the refresh mutation and retrieve
updated subscription fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-09-29 12:35:18 +00:00
Wu Yue 9f94d5c216 feat(core): support ai chat delete action (#13655)
<img width="411" height="205" alt="截屏2025-09-26 10 58 39"
src="https://github.com/user-attachments/assets/c3bce144-7847-4794-b766-5a3777cbc00d"
/>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- New Features
- Delete icon added to AI session history with tooltip and confirmation
prompt; deleting current session opens a new session.
- Session deletion wired end-to-end (toolbar → provider → backend) and
shows notifications.

- Improvements
- Cleanup now supports deleting sessions with or without a document ID
(document-specific or workspace-wide).
- UI tweaks for cleaner session item layout and safer click handling
(delete won’t trigger item click).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-09-27 11:58:58 +00:00