mirror of
https://github.com/toeverything/AFFiNE.git
synced 2026-07-12 07:36:42 +08:00
DarkSky
ca2462f987
#### PR Dependency Tree * **PR #14243** 👈 This tree was auto-generated by [Charcoal](https://github.com/danerwilliams/charcoal) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Batch management API for coordinated document mutations and change tracking. * New document accessors (IDs, state snapshots, change/delete set queries) and subscriber count. * **Chores** * Upgraded Rust edition across packages to 2024. * Repository-wide formatting, stylistic cleanups and test adjustments. * **Breaking Changes** * Removed the Node native bindings package and its JS/TS declarations and tests (no longer published/available). <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai -->
29 lines
688 B
Rust
29 lines
688 B
Rust
mod docx;
|
|
mod html;
|
|
mod pdf;
|
|
mod source;
|
|
mod text;
|
|
|
|
use std::io::{Read, Seek};
|
|
|
|
use super::*;
|
|
|
|
// modified from https://github.com/Abraxas-365/langchain-rust/tree/v4.6.0/src/document_loaders
|
|
pub trait Loader: Send + Sync {
|
|
fn load(self) -> LoaderResult<Vec<Document>>;
|
|
fn load_and_split<TS: TextSplitter + 'static>(self, splitter: TS) -> LoaderResult<Vec<Document>>
|
|
where
|
|
Self: Sized,
|
|
{
|
|
let docs = self.load()?;
|
|
Ok(splitter.split_documents(&docs)?)
|
|
}
|
|
}
|
|
|
|
pub use docx::DocxLoader;
|
|
pub use html::HtmlLoader;
|
|
pub use pdf::PdfExtractLoader;
|
|
pub use source::{LanguageParserOptions, SourceCodeLoader, get_language_by_filename};
|
|
pub use text::TextLoader;
|
|
pub use url::Url;
|