fix(server): frequent embedding (#13475)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- New Features
- Smarter embedding pipeline skips re-embedding when content hasn’t
changed; added content sanitization for embeddings and workspace content
retrieval.
- Bug Fixes
- Re-embedding now requires both a document update and the last
embedding being older than 10 minutes, reducing unnecessary work.
- Refactor
- Consolidated embedding preprocessing and moved sanitization utilities
into shared models; upserts now refresh stored content.
- Tests
- Expanded snapshot-based tests covering multiple time/age scenarios for
embedding decision logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
DarkSky
2025-08-12 09:45:41 +08:00
committed by GitHub
parent 125564b7d2
commit 65f679c4f0
10 changed files with 177 additions and 43 deletions
@@ -3,11 +3,14 @@ import { omit } from 'lodash-es';
import { z } from 'zod';
import type { AccessController } from '../../../core/permission';
import type { ChunkSimilarity, Models } from '../../../models';
import {
type ChunkSimilarity,
clearEmbeddingChunk,
type Models,
} from '../../../models';
import type { CopilotContextService } from '../context';
import type { ContextSession } from '../context/session';
import type { CopilotChatOptions } from '../providers';
import { clearEmbeddingChunk } from '../utils';
import { toolError } from './error';
export const buildDocSearchGetter = (