Commit Graph

38 Commits

Author SHA1 Message Date
DarkSky b530198a3b feat(server): improve indexer perf (#15512)
#### PR Dependency Tree


* **PR #15512** 👈

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Search now supports generation-based indexing with embedded and remote
providers.
* Added automatic search reconciliation and improved handling of
document, workspace, and permission changes.
* Added clearer search status errors for unavailable, syncing, unready,
or failed indexes.
* Added Manticore Search end-to-end support and provider-specific search
behavior.

* **Improvements**
* Search and aggregate pagination now report returned results and
continuation status more accurately.
* Improved permission filtering to prevent inaccessible documents from
appearing in results.
  * Admin provider selection now consistently enables indexing.

* **Documentation**
* Clarified search pagination, aggregation counts, provider
configuration, and end-to-end setup.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-23 17:54:49 +08:00
DarkSky c57004ea2c refactor(server): indexer & worker & sync perf (#15504) 2026-08-21 08:11:37 +08:00
DarkSky 998b255afd feat(server): refactor mail queue (#15204) 2026-07-07 08:38:16 +08:00
DarkSky 0a422aa158 feat(server): blob reconciliation (#15165)
#### PR Dependency Tree


* **PR #15165** 👈

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added automated backend maintenance for missing blob metadata
backfill, document-to-blob reference rebuilding, and unreferenced blob
cleanup planning/execution.
* Introduced scheduled batch processing (workspace-paged) and paginated
object-storage listing.
* **Bug Fixes**
* Improved reliability of object-storage reads by treating expected “not
found” results as non-errors.
* Strengthened blob/expired cleanup flows with runtime-driven batching
and reduced coupling to metadata synchronization.
* **Tests**
* Expanded unit and e2e coverage for partial blob metadata and updated
runtime/job cleanup test assertions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-29 00:02:38 +08:00
DarkSky 7123595831 chore: bump deps (#15059)
#### PR Dependency Tree


* **PR #15059** 👈

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Configurable minimum account age before new accounts can invite
members or create share links (default: 24 hours).
* Sign-in now returns and caches user info for improved session
handling.

* **Bug Fixes**
  * Queue handling accepts and resolves job IDs with special characters.
* Improved clipboard/rich-text caret handling and nested-list paste
reliability.
  * Calendar tests use dynamic current-month dates.
  * AI search returns explicit "No matching documents" when none found.
  * Auth session responses are explicitly non-cacheable.

* **Chores**
* Dependency and toolchain bumps; admin UI config/schema exposes the new
account-age setting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-01 20:13:59 +08:00
DarkSky b05c387f96 fix(server): mail test & retry (#15044)
#### PR Dependency Tree


* **PR #15044** 👈

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Stop sending notifications to disabled users; skip member invites when
workspace names contain URLs/domains
* Improve mail retry handling (per-recipient exhaustion, expiry, and
cache cleanup)
  * Make many email headers/lead lines more generic and consistent
  * Fail-safe workspace content parsing to avoid crashes

* **New Features**
* 24-hour signup protection for sharing, invites, and invite-link
creation
  * Job-queue: remove jobs by payload predicate

* **Tests**
* Expanded tests for mail jobs, SMTP hostname handling, payment
checkout, job-queue removal, and abuse-detection utilities
  * Updated test fixtures to set createdAt timestamps for new users

* **Chores**
  * Added required name input for test-email mutation
  * Database flush retry with deadlock detection/backoff

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/toeverything/AFFiNE/pull/15044?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-31 00:06:29 +08:00
DarkSky 0ccfacbc29 feat(docs): migrate bs docs 2026-04-29 17:23:23 +08:00
DarkSky d975bf46fb feat(server): improve calendar sync queue (#14783)
#### PR Dependency Tree


* **PR #14783** 👈

This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Configurable request timeout for calendar integrations.
* Calendar polling now enqueues per-subscription sync jobs (larger
batch) for improved throughput.

* **Bug Fixes / Improvements**
* Persisted next-sync timestamps and retry counts for more reliable
scheduling and retry behavior.
* Exponential backoff and webhook renewal now update scheduling
consistently.

* **Refactor**
* Calendar sync flow moved to a job-queue-driven design for better
concurrency and observability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-05 10:52:18 +08:00
DarkSky 046e126054 feat: bump typescript (#14507)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
  * Upgraded TypeScript toolchain to v5.9.3 across packages and tooling.
* Removed legacy ts-node and migrated developer tooling to newer
runtimes (tsx/SWC) where applicable.
* **Documentation**
* Updated developer CLI docs and runtime behavior notes to reflect the
new loader/runtime for running TypeScript files; no changes to public
APIs or end-user behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-02-24 13:22:46 +08:00
DarkSky 98e5747fdc feat: merge service 2026-02-13 21:52:11 +08:00
DarkSky bc03fab649 chore: cleanup logs 2025-12-31 12:31:58 +08:00
fengmk2 31a1841e25 chore(server): log removed job id (#12646)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **Chores**
	- Improved log messages to include job IDs when jobs are removed from the queue, enhancing traceability for users monitoring job activity.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-29 11:48:16 +00:00
fengmk2 9e5d132bd0 chore(server): log job start and finish (#12610)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **Refactor**
  - Improved visibility of job start and finish events by updating logging level, making these events more prominent in logs.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-28 09:51:37 +00:00
fengmk2 9abbfa3ab4 chore(server): print jobId (#12593)
Need to query the payload through job id for debugging

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **Tests**
  - Updated job metrics test to include an explicit job ID during execution.

- **Refactor**
  - Enhanced job execution to support an optional job ID, improving job tracking and logging.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-28 08:48:38 +00:00
darkskygit 9220b973c7 feat(server): increase embedding jobs concurrency & handle empty content after trim (#12574)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **Improvements**
  - Increased the default concurrency for background tasks, enhancing processing efficiency.
  - Improved handling of empty or unsupported documents to ensure consistent processing.
  - Optimized document filtering to exclude certain documents from processing, improving performance.

- **Bug Fixes**
  - Enhanced detection of empty document summaries, reducing errors during processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-27 14:28:34 +00:00
fengmk2 e3d63896bf chore(server): add job backoff strategies (#12499)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **New Features**
  - Implemented an exponential backoff retry strategy for job queues, resulting in progressively longer wait times between retry attempts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-26 02:33:58 +00:00
darkskygit 6f9361caee feat(server): trigger workspace embedding (#12328)
fix AI-127

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **New Features**
  - Added automated event handling for workspace updates and document embedding, streamlining document embedding workflows.
  - Introduced detection and queuing of documents needing embedding, excluding ignored documents.
- **Improvements**
  - Enhanced performance of embedding-related searches by filtering results at the database level.
  - Increased concurrency for embedding job processing to improve throughput.
- **Bug Fixes**
  - Improved error handling and fallback for missing document titles during embedding.
  - Added safeguards to skip invalid embedding jobs based on document identifiers.
- **Tests**
  - Expanded test coverage for document embedding and ignored document filtering.
  - Updated end-to-end tests to use dynamic content for improved reliability.
  - Added synchronization waits in document creation utilities to improve test stability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-20 05:16:45 +00:00
fengmk2 a34c3ea200 feat(server): auto index all workspaces to indexer (#12205)
close CLOUD-207

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **New Features**
  - Introduced automated periodic indexing of workspaces with a new job type and a scheduled cron job running every 30 seconds.
  - Added a unique sequential identifier (`sid`) and an "indexed" flag to workspaces to track indexing status.
- **Improvements**
  - Enhanced workspace indexing to handle missing workspaces and snapshots distinctly and selectively index documents.
  - Added ability to query workspaces after a given identifier with result limits.
- **Bug Fixes**
  - Improved error handling and logging during workspace indexing operations.
- **Tests**
  - Expanded test coverage for workspace indexing and auto-indexing, including scheduling and edge cases.
- **Chores**
  - Updated data models and schema to support new workspace fields and indexing features.
  - Enhanced mock data utilities to allow custom timestamps.
  - Improved type safety and flexibility in document snapshot retrieval.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-05-19 04:39:38 +00:00
fengmk2 afa984da54 feat(server): handle workspace doc updates (#11937) 2025-05-14 14:52:41 +00:00
fengmk2 ff133d1267 refactor(server): reduce server logs (#11931)
close CLOUD-201
2025-04-23 11:41:48 +00:00
forehalo 8e1cbc4c5b test(server): do not consume job in test (#11592) 2025-04-10 07:02:07 +00:00
forehalo 6939e80827 chore(server): never reuse cls id for job handler (#11449) 2025-04-03 12:26:02 +00:00
forehalo 35bea20b80 chore(server): adjust job config (#11405) 2025-04-02 09:48:46 +00:00
forehalo 85d176ce6f chore(server): customize config merge logic (#11400) 2025-04-02 09:48:45 +00:00
fengmk2 432634c5a3 feat(server): enable job queue log (#11390) 2025-04-02 09:34:13 +00:00
forehalo 6276732efc chore(server): reschedule doc merging jobs (#11318) 2025-04-01 10:57:54 +00:00
forehalo 0ea38680fa refactor(server): config system (#11081) 2025-03-27 12:32:28 +00:00
fengmk2 d7b3dc683b feat(server): send invitation notification (#10219)
close PD-2307 CLOUD-150
2025-03-24 02:27:23 +00:00
fengmk2 55a60906a5 test(server): add MockJobQueue (#11032) 2025-03-21 04:08:29 +00:00
forehalo 21c4a29f55 refactor(server): mail service (#10934) 2025-03-19 17:00:19 +00:00
darkskygit b099546164 feat(server): copilot job models (#10732) 2025-03-19 14:34:14 +00:00
forehalo d50860eee2 fix(server): use updowncounter by default (#10482) 2025-02-27 10:15:13 +00:00
liuyi b3821ad619 fix(server): avoid global rejection when event handler errors (#10467) 2025-02-27 06:25:46 +00:00
fengmk2 b4097aef8e refactor(server): move bin content parser to doc reader (#10302) 2025-02-20 07:19:48 +00:00
fengmk2 ff0ce1a962 fix(server): remove job on complete (#10305) 2025-02-20 04:39:05 +00:00
liuyi 521ee9d374 fix(server): doc owner and default role permission (#10281) 2025-02-19 08:29:46 +00:00
liuyi 294002101d fix(server): wrong queue options for worker executor (#10267) 2025-02-19 04:44:23 +00:00
forehalo cb895d4cb0 feat(server): job system (#10134) 2025-02-18 05:41:57 +00:00