close AI-186
#### PR Dependency Tree
* **PR #12811** 👈
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added a new tool enabling users to read document content and metadata
within a workspace, with enforced access control.
- **Improvements**
- Updated tool interfaces and outputs to support the document reading
functionality seamlessly.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced support for the Morph provider in the copilot module,
enabling integration with the Morph LLM API.
- Added a new document editing tool that allows users to propose and
apply edits to existing documents via AI assistance.
- **Configuration**
- Added configuration options for the Morph provider in both backend and
admin interfaces.
- **Enhancements**
- Expanded tool support to include document editing capabilities within
the copilot feature set.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Improved document reranking with a more streamlined and accurate
scoring system.
* Enhanced support for binary ("yes"/"no") document relevance judgments.
* **Improvements**
* Simplified user prompts and output formats for reranking tasks, making
results easier to interpret.
* Increased reliability and consistency in document ranking results.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<img width="775" alt="截屏2025-06-26 16 17 05"
src="https://github.com/user-attachments/assets/ed6bcae3-94af-4eb1-81e8-710f36ef5e46"
/>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Introduced a tool to crawl web pages and extract key information.
* Added a visual component to display tool call failures in the AI
interface.
* Enhanced error reporting for document and web search tools with
structured error messages.
* **Improvements**
* Updated error handling across AI tools and components for more
consistent and informative feedback.
* Default values added for tool card components to improve reliability
and display.
* **Bug Fixes**
* Improved handling of error and empty states in web crawl and web
search result displays.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close AI-185
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a keyword-based document search tool, allowing users to
search for relevant documents within their workspace using keywords.
- Search results include document titles, summaries, and direct links,
enhancing document discovery and navigation.
- **Bug Fixes**
- None.
- **Tests**
- Added new tests to verify document search by IDs and by keywords,
ensuring accurate and reliable search functionality.
- **Documentation**
- None.
- **Chores**
- Updated configuration file organization for improved clarity; no
changes to functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#### PR Dependency Tree
* **PR #12867**
* **PR #12863**
* **PR #12837** 👈
This tree was auto-generated by
[Charcoal](https://github.com/danerwilliams/charcoal)
Close [AI-193](https://linear.app/affine-design/issue/AI-193)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added support for streaming structured AI chat responses as objects,
enabling richer and more interactive chat experiences.
- Chat messages now include a new field displaying structured stream
objects, such as reasoning steps, text deltas, tool calls, and tool
results.
- GraphQL APIs and queries updated to expose these structured streaming
objects in chat histories.
- Introduced a new streaming chat endpoint for object-based responses.
- **Bug Fixes**
- Improved error handling for streaming responses to ensure more robust
and informative error reporting.
- **Refactor**
- Centralized and streamlined session preparation and streaming logic
for AI chat providers.
- Unified streaming setup across multiple AI model providers.
- **Tests**
- Extended test coverage for streaming object responses to ensure
reliability and correctness.
- **Documentation**
- Updated type definitions and schemas to reflect new streaming object
capabilities in both backend and frontend code.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: DarkSky <25152247+darkskygit@users.noreply.github.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated Gemini model identifiers from preview versions to their latest
stable versions across prompts and provider configurations. This ensures
continued compatibility and access to the latest Gemini 2.5 Flash and
Pro models. No changes to user-facing features or prompt content.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Updated document-related tools in chat prompts, including new options
for document keyword and semantic search.
- **Refactor**
- Renamed and reorganized document search tools for improved clarity in
tool selection and results display.
- **Bug Fixes**
- Ensured tool identifiers and results are consistent across chat and
search features.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Refactor**
- Improved the handling of attachments in chat messages for more efficient processing of images and files without impacting user experience.
- **Chores**
- Added internal logging to enhance monitoring of AI model interactions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Support [AI-59](https://linear.app/affine-design/issue/AI-59)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Updated the default AI model for chat prompts to use Claude Sonnet 4.
- **Bug Fixes**
- Improved model selection logic to better support reasoning features across more AI models.
- **Tests**
- Enhanced test cases with consistent instructions for response length.
- Skipped certain chat-related tests to refine test suite stability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-14
fix AI-17
fix AI-39
fix AI-112
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Expanded and reorganized prompt options for text and image actions, adding new prompts for image generation, style conversions, upscaling, background removal, and sticker creation.
- Enhanced image editing capabilities with direct support for image attachments in prompts.
- **Improvements**
- Updated prompt names and descriptions to be more user-friendly and descriptive.
- Simplified and clarified prompt selection and image processing workflows with improved default behaviors.
- Better organization of prompts through clear grouping and categorization.
- **Bug Fixes**
- Improved validation and handling of image attachments during editing requests.
- **Refactor**
- Internal code restructuring of prompts and provider logic for clarity and maintainability without affecting user workflows.
- Refined message handling and content merging logic to ensure consistent prompt processing.
- Adjusted image attachment rendering logic for improved display consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Support [AI-82](https://linear.app/affine-design/issue/AI-82).
Added a `TextStreamParser` class to standardize formatting of different types of AI stream chunks across providers.
### What changed?
- Created a new `TextStreamParser` class in `utils.ts` that handles formatting of various chunk types (text-delta, reasoning, tool-call, tool-result, error)
- Refactored the Anthropic, Gemini, and OpenAI providers to use this shared parser instead of duplicating formatting logic
- Added comprehensive tests for the new `TextStreamParser` class, including tests for individual chunk types and sequences of chunks
- Defined a common `AITools` type to standardize tool interfaces across providers
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Enhanced formatting and structure for streamed AI responses, including improved handling of callouts, web search, and web crawl results.
- **Refactor**
- Streamlined and unified the processing of streamed AI response chunks across providers for more consistent output.
- **Bug Fixes**
- Improved error handling and display for streamed responses.
- **Tests**
- Added comprehensive tests to ensure correct formatting and handling of various streamed message types.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Support [AI-59](https://linear.app/affine-design/issue/AI-59)
Deprecated use of the perplexity model. Makes chat and search use the same prompt content.
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
- **New Features**
- Added support for two new AI models: Claude Opus 4 and Claude Sonnet 4, enabling enhanced text and image input capabilities.
- Introduced a new chat prompt with detailed instructions for specialized AI interactions within AFFiNE.
- **Refactor**
- Improved prompt management for AI chat and search features by centralizing shared settings for better consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Close [AI-125](https://linear.app/affine-design/issue/AI-125)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added new provider configurations `geminiVertex` and `anthropicVertex` for Google Vertex AI in backend schema, provider classes, and admin config.
- Introduced `GeminiVertexProvider` and `AnthropicVertexProvider` classes supporting Vertex AI models with specific capabilities.
- Expanded model options for transcription prompts with newer Gemini models.
- Re-exported provider modules to include Vertex AI variants.
- **Improvements**
- Extended provider architecture to support separate Vertex AI configurations and models.
- Updated test setup to replace deprecated provider references with new Vertex variants.
- Consolidated environment variables for server testing with a single `SERVER_CONFIG`.
- **Bug Fixes**
- Updated mock models and import references in tests to align with new provider classes.
- **Chores**
- Added `@ai-sdk/google-vertex` dependency for Vertex AI support.
- Updated dependency list to include `@ai-sdk/google-vertex`.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Close [AI-125](https://linear.app/affine-design/issue/AI-125)
What Changed?
- Add `gemini-2.5-flash-preview-04-17` model
- Add `thinkingConfig` provider options
- Add callout syntax highlighting
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added support for the "Gemini 2.5 Flash" model and updated the "Gemini 2.5 Pro" model to a newer version.
- Enhanced streaming responses to better format reasoning outputs and provide clearer callouts in AI-generated content.
- **Bug Fixes**
- Improved audio transcription prompts in test cases for more accurate and explicit testing.
- **Documentation**
- Expanded citation instructions for AI chat responses, including examples for multiple citations.
- **Chores**
- Updated the "@ai-sdk/google" dependency to a newer version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Improved input validation and error reporting for chat messages, attachments, and embeddings, with clearer error messages for invalid inputs.
- Enhanced support for multimodal messages, including attachments such as images or audio.
- **Refactor**
- Unified and streamlined parameter validation across AI providers, resulting in more consistent behavior and error handling.
- Centralized parameter checks into a common provider layer, removing duplicate validation code from individual AI providers.
- **Tests**
- Simplified and consolidated audio transcription test stubs for better maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-4
fix AI-18
better provider/model choose to allow fallback to similar models (e.g., self-hosted) when the provider is not fully configured
split functions of different output types
Close [AI-86](https://linear.app/affine-design/issue/AI-86)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced AI model switching in chat, allowing users to select from multiple AI models during conversations.
- Added a floating menu for easy AI model selection within the chat interface.
- Enabled visibility of the AI model switcher through a new experimental feature flag, configurable in workspace settings (canary builds only).
- **Enhancements**
- Improved session management in the chat panel for smoother model switching and state handling.
- Updated localization to support the new AI model switch feature in settings.
- **Bug Fixes**
- None.
- **Chores**
- Registered new components and services to support AI model switching functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Close [AI-126](https://linear.app/affine-design/issue/AI-126)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Introduced a new web crawling tool, allowing users to extract live content from specific web pages in addition to traditional web search.
- **Improvements**
- Enhanced error handling for web search and web crawl operations, providing clearer failure messages.
- Updated terminology in AI prompts and user-facing messages to reflect the new web search/crawl capabilities.
- Improved formatting of web search and crawl results for better readability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Close [AI-116](https://linear.app/affine-design/issue/AI-116)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added support for specifying alternative AI models in chat prompts, enabling users to select from multiple available models.
- Expanded AI model options with new additions: 'gpt-4.1', 'o3', and 'claude-3-5-sonnet-20241022'.
- **Enhancements**
- Users can now optionally choose a specific AI model during chat interactions.
- Prompts and chat sessions reflect and support selection of alternative models where applicable.
- **Bug Fixes**
- Improved handling of prompt configuration defaults for better reliability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Close [AI-75](https://linear.app/affine-design/issue/AI-75)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Improved formatting for AI-generated reasoning and web search results, displaying them as callout blocks for enhanced readability.
- Expanded support for rendering callout blocks in the text renderer.
- **Style**
- Adjusted layout to ensure callout blocks retain appropriate spacing and appearance.
- **Refactor**
- Simplified and unified the integration of web search tools for AI providers.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Close [AI-85](https://linear.app/affine-design/issue/AI-85)
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Enhanced AI chat experience with improved web search integration, including a new "AUTO" mode that allows the AI to search the web only when needed.
- Updated AI model for chat to "o4-mini" for improved performance.
- **Improvements**
- Refined instructions to reduce hallucinations and ensure the AI admits uncertainty when unsure.
- Enhanced streaming responses to display web search queries and results more clearly.
- Updated web search tool modes for more accurate and flexible search behavior.
- Centralized AI provider options and improved tool selection logic for better response quality.
- **Bug Fixes**
- Improved handling of tool calls and reasoning steps in AI chat responses.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Enabled web search capabilities in the "Chat With AFFiNE AI" prompt, allowing users to access web-based information directly within the chat.
- **Improvements**
- Enhanced support for multiple tools in AI chat prompts, paving the way for future tool integrations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Support [AI-60](https://linear.app/affine-design/issue/AI-60).
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Refactor**
- Updated naming conventions for the web search tool to ensure consistency across AI chat features.
- Simplified internal handling of web search tool options for a more streamlined experience.
- Unified parameter naming from "mustSearch" to "webSearch" across AI chat inputs and actions.
- **Chores**
- Removed unused configuration options related to web search from prompt settings.
- Added support for enabling or disabling web search via new parameters in chat requests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved handling of messages without attachments, ensuring a placeholder is shown when content is empty and attachments are omitted.
- **New Features**
- Added the ability to selectively include or exclude attachments in message processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Close [AI-40](https://linear.app/affine-design/issue/AI-40).
### What changed?
- Add anthropic provider.
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added support for Anthropic as a new provider option in Copilot, enabling integration with Anthropic's text generation models.
- Users can now configure Anthropic provider settings, including API key management, through the admin interface.
- **Chores**
- Updated dependencies to include the Anthropic SDK for backend operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fix AI-88
<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit
- **New Features**
- Added support for the new image generation model "gpt-image-1" in both backend and frontend.
- Updated available prompt options to include "gpt-image-1" for image creation actions.
- **Improvements**
- Set a 5-minute timeout for image generation requests to improve reliability.
- **Tests**
- Expanded test coverage to validate image generation using the new "gpt-image-1" prompt.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
fengmk2
