Commit Graph

22 Commits

Author SHA1 Message Date
DarkSky
059d3aa04a feat: improve native 2026-02-01 20:15:41 +08:00
DarkSky
f373e08583 feat: refactor doc write in native (#14272) 2026-01-18 16:31:12 +08:00
realies
0da91e406e feat(server): add document write tools for mcp (#14245)
## Summary

This PR adds write capabilities to AFFiNE's MCP (Model Context Protocol)
integration, enabling external tools (Claude, GPT, etc.) to create and
modify documents programmatically.

**New MCP Tools:**
- `create_document` - Create new documents from markdown content
- `update_document` - Update document content using structural diffing
for minimal changes (preserves document history and enables real-time
collaboration)

**Implementation:**
- `markdown_to_ydoc.rs` - Converts markdown to AFFiNE-compatible y-octo
binary format
- `markdown_utils.rs` - Shared markdown parsing utilities (used by both
ydoc-to-md and md-to-ydoc)
- `update_ydoc.rs` - Structural diffing implementation for updating
existing documents
- `DocWriter` service - TypeScript service for document operations
- Exposes `markdownToDocBinary` and `updateDocBinary` via napi bindings

**Supported Markdown Elements:**
- Headings (H1-H6)
- Paragraphs
- Bullet lists and numbered lists
- Code blocks (with language detection)
- Blockquotes
- Horizontal dividers
- Todo items (checkboxes)

**y-octo Changes:**
This PR reverts the y-octo sync (ca2462f, a5b60cf) which introduced a
concurrency bug causing hangs when creating documents with many nested
block structures. It also ports the improved `get_node_index` binary
search fix from upstream that prevents divide-by-zero panics when
decoding documents.

## Test Results 

### Unit Tests (47/47 passing)

| Test Suite | Tests | Status |
|------------|-------|--------|
| markdown_to_ydoc | 16/16 |  Pass |
| markdown_utils | 11/11 |  Pass |
| update_ydoc | 13/13 |  Pass |
| delta_markdown | 2/2 |  Pass |
| affine (doc parser) | 5/5 |  Pass |

### End-to-End MCP Testing 

Tested against local AFFiNE server with real MCP client requests:

| Tool | Result | Notes |
|------|--------|-------|
| `tools/list` |  Pass | Returns all 5 tools with correct schemas |
| `create_document` |  Pass | Successfully created test documents |
| `update_document` |  Pass | Successfully updated documents with
structural diffing |
| `read_document` |  Pass | Existing tool, works correctly |
| `keyword_search` |  Pass | Existing tool, works correctly |

**E2E Test Details:**
- Started local AFFiNE server with PostgreSQL, Redis, and Manticore
- Created test user and workspace via seed/GraphQL
- Verified MCP endpoint at `/api/workspaces/:workspaceId/mcp`
- Tested JSON-RPC calls with proper SSE streaming
- Confirmed documents are stored and indexed correctly (verified via
server logs)

## Test Plan
- [x] All Rust unit tests pass (47 tests)
- [x] Native bindings build successfully (release mode)
- [x] Document creation via MCP works end-to-end
- [x] Document update via MCP works end-to-end
- [x] CodeRabbit feedback addressed
- [ ] Integration testing with Claude/GPT MCP clients

Closes #14161

---

**Requested by:** @realies  
**Key guidance from:** @darkskygit (use y-octo instead of yjs for memory
efficiency)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Create documents from Markdown: generate new documents directly from
Markdown content with automatic title extraction
* Update documents with Markdown: modify existing documents using
Markdown as the source with automatic diff calculation for efficient
updates
* Copilot integration: new tools for document creation and updates
through Copilot's interface

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-16 20:57:24 +08:00
DarkSky
ca2462f987 feat(native): sync yocto codes (#14243)
#### PR Dependency Tree


* **PR #14243** 👈

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Batch management API for coordinated document mutations and change
tracking.
* New document accessors (IDs, state snapshots, change/delete set
queries) and subscriber count.

* **Chores**
  * Upgraded Rust edition across packages to 2024.
  * Repository-wide formatting, stylistic cleanups and test adjustments.

* **Breaking Changes**
* Removed the Node native bindings package and its JS/TS declarations
and tests (no longer published/available).

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-11 06:08:33 +08:00
DarkSky
ff2e96d847 feat: database indexing support (#14181) 2025-12-30 05:23:09 +08:00
DarkSky
321965a424 feat: impl text delta support (#14132) 2025-12-22 03:16:16 +08:00
DarkSky
844b9d9592 feat(server): impl native reader for server (#14100) 2025-12-14 00:28:43 +08:00
DarkSky
b7ac7caab4 chore(server): improve transcript stability (#13821)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Enhanced audio/video detection for MP4 files to better distinguish
audio-only vs. video.

* **Dependencies**
* Added MP4 parsing dependency and updated AI provider libraries
(Anthropic, Google, OpenAI, etc.).

* **Bug Fixes**
  * Tightened authentication state validation for magic-link/OTP flows.
* Stricter space-join validation to reject invalid client
types/versions.
  * Improved transcript entry deduplication and data handling.

* **API**
* Transcript submit payload now requires infos and removes deprecated
url/mimeType fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-10-29 17:48:15 +08:00
DarkSky
1b859a37c5 feat: improve attachment headers (#13709)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Safer, consistent file downloads with automatic attachment headers and
filenames.
- Smarter MIME detection for uploads (avatars, workspace blobs, Copilot
files/transcripts).
  - Sensible default buffer limit when reading uploads.

- **Bug Fixes**
- Prevents risky content from rendering inline by forcing downloads and
adding no‑sniff protection.
- More accurate content types when original metadata is missing or
incorrect.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-10-09 08:04:18 +00:00
DarkSky
072557eba1 feat(server): adapt gpt5 (#13478)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- New Features
- Added GPT-5 family and made GPT-5/-mini the new defaults for Copilot
scenarios and prompts.

- Bug Fixes
- Improved streaming chunk formats and reasoning/text semantics,
consistent attachment mediaType handling, and more reliable reranking
via log-prob handling.

- Refactor
- Unified maxOutputTokens usage; removed per-call step caps and migrated
several tools to a unified inputSchema shape.

- Chores
- Upgraded AI SDK dependencies and bumped an internal dependency
version.

- Tests
- Updated mocks and tests to reference GPT-5 variants and new stream
formats.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-08-13 02:32:15 +00:00
darkskygit
13fa4f922a fix(server): token calculate (#12667) 2025-06-04 07:09:33 +00:00
Brooooooklyn
d02aa8c7e0 fix(native): opt out napi-derive noop feature (#12686)
It would cause the napi-derive not work as expect in workspace level

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **Refactor**
  - Improved internal handling and type definitions for document parsing, resulting in clearer and more maintainable data structures.
- **Chores**
  - Introduced a new feature flag for mobile native builds, enabling conditional compilation for enhanced flexibility across Android and iOS.
  - Updated build scripts to support the new feature flag for both Android and iOS platforms.
  - Updated iOS app dependencies to newer versions, including Apollo iOS, ChidoriMenu, and swift-collections, and removed SQLite.swift.
- **Tests**
  - Enhanced Rust linting and testing workflows to run selectively across workspace packages with the new feature flag enabled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-06-03 06:46:55 +00:00
Brooooooklyn
e1ce42a6fc feat(native): upgrade NAPI-RS to 3.0.0 beta (#12652)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **New Features**
	- Added a default export for the native binding in the frontend native module, allowing easier imports.

- **Refactor**
	- Streamlined and updated Rust-to-JavaScript type conversions and lifetime handling for improved safety and consistency.
	- Improved object and array construction in Rust modules for more idiomatic usage.
	- Simplified boolean and null value handling in JavaScript interop layers.

- **Chores**
	- Upgraded several dependencies and development tools to newer versions across backend, frontend, and common packages.
	- Updated build scripts for the frontend native package to simplify commands.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-29 16:09:32 +00:00
forehalo
dc7cd0487b refactor(server): decrypt license with provided aes key (#12570)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **New Features**
  - Added support for a new AES key for license management, improving license encryption and decryption processes.

- **Bug Fixes**
  - Improved error messages and handling when activating expired or invalid licenses.

- **Refactor**
  - Updated license decryption logic to use a fixed AES key instead of deriving one from the workspace ID.
  - Added validation for environment variable values to prevent invalid configurations.

- **Tests**
  - Enhanced license-related tests to cover new key usage and updated error messages.
  - Updated environment variable validation tests with clearer error messages.

- **Chores**
  - Updated environment variable handling for improved consistency.
  - Set production environment variable explicitly in build configuration.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-27 11:54:28 +00:00
forehalo
93e01b4442 feat(server): support installable license (#12181)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **New Features**
  - Added support for installing self-hosted team licenses via encrypted license files.
  - Introduced a new "Onetime" license variant for self-hosted environments.
  - Added a GraphQL mutation to upload and install license files.
  - License details now display the license variant.

- **Bug Fixes**
  - Improved error messages for license activation and expiration, including dynamic reasons.

- **Localization**
  - Updated and improved license-related error messages for better clarity.

- **Tests**
  - Added comprehensive end-to-end tests for license installation scenarios.

- **Chores**
  - Enhanced environment variable handling and public key management for license verification.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-09 04:16:05 +00:00
darkskygit
d8373f66e7 feat(server): context awareness for copilot (#9611)
fix PD-2167
fix PD-2169
fix PD-2190
2025-03-13 11:44:55 +00:00
darkskygit
d25b216311 feat(server): adapt doc loader for server native (#9942) 2025-02-26 08:05:20 +00:00
darkskygit
1c2b23b160 feat(native): split package (#8853) 2024-11-18 13:18:16 +00:00
Brooooooklyn
fe1eefdbb2 feat: init renderer server (#8088) 2024-09-10 04:03:59 +00:00
Brooooooklyn
c7ddd679fd feat(server): use native tokenizer impl (#6960)
### Benchmark

`yarn workspace @affine/server-native bench`

```
┌─────────┬────────────┬─────────┬────────────────────┬──────────┬─────────┐
│ (index) │ Task Name  │ ops/sec │ Average Time (ns)  │ Margin   │ Samples │
├─────────┼────────────┼─────────┼────────────────────┼──────────┼─────────┤
│ 0       │ 'tiktoken' │ '5'     │ 176932518.76000002 │ '±4.71%' │ 100     │
│ 1       │ 'native'   │ '16'    │ 61041597.51000003  │ '±0.60%' │ 100     │
└─────────┴────────────┴─────────┴────────────────────┴──────────┴─────────┘
```
2024-05-16 07:55:10 +00:00
Brooooooklyn
81729703d9 chore(server): use native file-type implementation (#6686) 2024-04-29 04:46:26 +00:00
Brooooooklyn
fed2503782 refactor(server): rename @affine/storage to @affine/server-native (#6682)
- Close https://github.com/toeverything/AFFiNE/issues/6680
2024-04-29 02:14:20 +00:00