Commit Graph

8 Commits

Author SHA1 Message Date
Francisco Jiménez 0b1a44863f feat(editor): add obsidian vault import support (#14593)
fix #14592 

### Description
> 🤖 **Note:** The code in this Pull Request were developed with the
assistance of AI, but have been thoroughly reviewed and manually tested.

> I noticed there's a check when opening an issue that asks _"Is your
content generated by AI?"_, so I mention it here in case it's a deal
breaker. If so I understand, you can close the PR, just wanted to share
this in case it's useful anyways.

This PR introduces **Obsidian Vault Import Support** to AFFiNE. 

Previously, users migrating from Obsidian had to rely on the generic
Markdown importer, which often resulted in broken cross-links, missing
directory structures, and metadata conflicts because Obsidian relies
heavily on proprietary structures not supported by standard Markdown.

This completely new feature makes migrating to AFFiNE easy.

**Key Features & Implementations:**

1. **Vault (Directory) Selection**
- Utilizes the `openDirectory` blocksuite utility in the import modal to
allow users to select an entire folder directly from their filesystem,
maintaining file context rather than forcing `.zip` uploads.

2. **Wikilink Resolution (Two-Pass Import)**
- Restructured the `importObsidianVault` process into a two-pass
architecture.
- **Pass 1:** Discovers all files, assigns new AFFiNE document IDs, and
maps them efficiently (by title, alias, and filename) into a
high-performance hash map.
- **Pass 2:** Processes the generic markdown AST and correctly maps
custom `[[wikilinks]]` to the actual pre-registered AFFiNE blocksuite
document IDs via `obsidianWikilinkToDeltaMatcher`.
- Safely strips leading emojis from wikilink aliases to prevent
duplicated page icons rendering mid-sentence.

3. **Emoji Metadata & State Fixes**
- Implemented an aggressive, single-pass RegExp to extract multiple
leading/combining emojis (`Emoji_Presentation` / `\ufe0f`) from H1
headers and Frontmatter. Emojis are assigned specifically to the page
icon metadata property and cleanly stripped from the visual document
title.
- Fixed a core mutation bug where the loop iterating over existing
`docMetas` was aggressively overwriting newly minted IDs for the current
import batch. This fully resolves the issue where imported pages
(especially re-imports) were incorrectly flagged as `trashed`.
   - Enforces explicit `trash: false` patch instructions.

4. **Syntax Conversion**
- Implemented conversion of Obsidian-style Callouts (`> [!NOTE] Title`)
into native AFFiNE block formats (`> 💡 **Title**`).
- Hardened the `blockquote` parser so that nested structures (like `> -
list items`) are fully preserved instead of discarded.

### UI Changes
- Updated the Import Modal to include the "Import Obsidian Vault" flow
utilizing the native filesystem directory picker.
- Regenerated and synced `i18n-completenesses.json` correctly up to 100%
across all supported locales for the new modal string additions.

### Testing Instructions
1. Navigate to the Workspace sidebar and click "Import".
2. Select "Obsidian" and use the directory picker to define a
comprehensive Vault folder.
3. Validate that cross-links between documents automatically resolve to
their specific AFFiNE instances.
4. Validate documents containing leading Emojis display exactly one
Emoji (in the page icon area), and none duplicated in the actual title
header.
5. Validate Callouts are rendered cleanly and correctly, and no
documents are incorrectly marked as "Trash".


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Import Obsidian vaults with wikilink resolution, emoji/title
preservation, asset handling, and automatic document creation.
* Folder-based imports via a Directory Picker (with hidden-input
fallback) integrated into the import dialog.

* **Localization**
  * Added Obsidian import label and tooltip translations.

* **Tests**
* Added end-to-end tests validating Obsidian vault import and asset
handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: DarkSky <25152247+darkskygit@users.noreply.github.com>
Co-authored-by: DarkSky <darksky2048@gmail.com>
2026-03-17 00:49:17 +08:00
DarkSky ac7a95e708 feat: misc optimizations (#14257)
fix #13798
2026-01-15 03:47:22 +08:00
德布劳外 · 贾贵 b53b4884cf refactor(core): align markdown conversion logic (#13254)
## Refactor

Align the Markdown conversion logic across all business modules:
1. frontend/backend apply: doc to markdown
2. insert/import markdown: use `markdownAdapter.toDoc`

> CLOSE AI-328 AI-379 AI-380

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Clarified instructions and provided an explicit example for correct
list item formatting in the markdown editing tool.

* **Bug Fixes**
* Improved markdown parsing for lists, ensuring correct indentation and
handling of trailing newlines.
* Cleaned up markdown snapshot test files by removing redundant blank
lines for better readability.

* **Refactor**
* Updated markdown conversion logic to use a new parsing approach for
improved reliability and maintainability.
* Enhanced markdown generation method for document snapshots with
improved error handling.
* Refined markdown-to-snapshot conversion with more robust document
handling and snapshot extraction.

* **Chores**
* Added a new workspace dependency for enhanced markdown parsing
capabilities.
* Updated project references and workspace dependencies to include the
new markdown parsing package.

* **Tests**
* Temporarily disabled two markdown-related tests due to parse errors in
test mode.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-07-21 10:35:13 +00:00
fengmk2 d80bfac1d2 feat(server): parse ydoc to ai editable markdown format (#12846)
close AI-213









#### PR Dependency Tree


* **PR #12846** 👈
  * **PR #12811**

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Introduced support for AI-editable blocks in document parsing,
allowing blocks to include metadata for AI-based editing.
  - Added rendering for todo list items with markdown checkbox syntax.
- Unsupported block types are now marked with placeholders in the parsed
output.

- **Tests**
- Added new test cases and snapshots to verify parsing behavior with
AI-editable content enabled.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-06-19 11:27:04 +08:00
Peng Xiao f4c20056a0 feat(core): moving in affine-reader doc parsers (#12840)
fix AI-191

#### PR Dependency Tree


* **PR #12840** 👈

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Introduced the ability to convert rich text documents into Markdown,
supporting a wide range of content types such as headings, lists,
tables, images, code blocks, attachments, and embedded documents.
- Added support for parsing collaborative document structures and
rendering them as structured Markdown or parsed representations.
- Enhanced handling of database and table blocks, including conversion
to Markdown tables with headers and cell content.

- **Documentation**
  - Added a README noting the use of a forked Markdown converter.

- **Tests**
  - Added new test coverage for document parsing features.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->


#### PR Dependency Tree


* **PR #12840** 👈

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
2025-06-17 16:32:11 +08:00
EYHN 04f3d88e2c feat(nbstore): add more data to indexer (#12815)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Improved display of image and attachment blocks by ensuring image
captions are shown as content for images, while attachment names remain
as content for attachments.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->


#### PR Dependency Tree


* **PR #12815** 👈

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

---------

Co-authored-by: fengmk2 <fengmk2@gmail.com>
2025-06-13 18:30:57 +08:00
fengmk2 39cb1afedb fix(server): limit rootDoc snapshot size (#12625)
close CLOUD-225

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **New Features**
  - Added support for reading document blocks without requiring a workspace or root document snapshot.
- **Bug Fixes**
  - Improved handling of large workspace snapshots by skipping them when they exceed 10MB.
- **Tests**
  - Introduced new test cases to cover scenarios where root or workspace snapshots are absent.
  - Expanded snapshot tests for document block reading.
- **Refactor**
  - Updated several function signatures to make root and workspace snapshot parameters optional for greater flexibility.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-29 05:32:31 +00:00
fengmk2 6792c3e656 feat(common): add blocksuite reader lib (#11955)
close CLOUD-202
2025-05-14 14:52:41 +00:00