Haystack 2.x Retrievers API 深度解析从 BM25、向量检索到句子窗口与自动合并【免费下载链接】haystackOpen-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.项目地址: https://gitcode.com/GitHub_Trending/ha/haystack本篇技术指南以 Haystack 2.x 的 Retrievers API 文档docs-website/reference_versioned_docs/version-2.18/haystack-api/retrievers_api.md为核心骨架系统讲解AutoMergingRetriever、InMemoryBM25Retriever、InMemoryEmbeddingRetriever、FilterRetriever与SentenceWindowRetriever五大组件的设计原理、完整参数语义、代码级调用链与实战用法。读完本文你将能够根据检索场景关键词、语义、过滤、父子文档合并、上下文窗口扩展正确选型并组装可运行的 Haystack 检索流水线。检索器在 Haystack 中的定位在 Haystack 中Retriever检索器是一个扫过 Document Store、返回与查询相关的候选文档集合的组件。它与Reader等组件的核心区别在于检索器只负责召回候选不做精读与答案生成它通常位于 RAG 流水线的前端将召回结果交给后续的 Prompt Builder、Generator 或 Evaluator 消费。从当前仓库源码看所有内置检索器都位于 haystack/components/retrievers/并通过component装饰器注册为可序列化、可连线的管道组件auto_merging_retriever.py —AutoMergingRetrieverin_memory/bm25_retriever.py —InMemoryBM25Retrieverin_memory/embedding_retriever.py —InMemoryEmbeddingRetrieverfilter_retriever.py —FilterRetrieversentence_window_retriever.py —SentenceWindowRetriever此外还有MultiQueryEmbeddingRetriever、MultiQueryTextRetriever、MultiRetriever、TextEmbeddingRetriever等组件见 haystack/components/retrievers/init.py本文聚焦 API 文档中展开讲解的五个组件。InMemoryBM25Retriever基于关键词的经典检索InMemoryBM25Retriever使用 BM25 关键词算法检索与查询最相似的文档必须搭配InMemoryDocumentStore使用。最小可用示例API 文档给出的最小示例完整可运行from haystack import Document from haystack.components.retrievers.in_memory import InMemoryBM25Retriever from haystack.document_stores.in_memory import InMemoryDocumentStore docs [ Document(contentPython is a popular programming language), Document(contentpython ist eine beliebte Programmiersprache), ] doc_store InMemoryDocumentStore() doc_store.write_documents(docs) retriever InMemoryBM25Retriever(doc_store) result retriever.run(queryProgrammiersprache) print(result[documents])构造参数与底层校验构造函数签名见 in_memory/bm25_retriever.pydef __init__(document_store: InMemoryDocumentStore, filters: Optional[dict[str, Any]] None, top_k: int 10, scale_score: bool False, filter_policy: FilterPolicy FilterPolicy.REPLACE)参数默认值语义document_store必填搜索目标必须是InMemoryDocumentStore实例否则抛TypeError源码第 70-71 行filtersNone用于缩小搜索空间的过滤字典元数据过滤top_k10返回的最大文档数必须 0否则抛ValueError源码第 75-76 行scale_scoreFalse为True时将得分归一化到 0-11 表示极相关为False返回原始相似度得分filter_policyREPLACE过滤策略见下文FilterPolicy 详解run 与 run_async运行期覆盖初始化参数run方法签名component.output_types(documentslist[Document]) def run(query: str, filters: Optional[dict[str, Any]] None, top_k: Optional[int] None, scale_score: Optional[bool] None)run_async拥有完全一致的签名用于异步执行。底层调用链非常直接源码第 146-153 行先通过apply_filter_policy(self.filter_policy, self.filters, filters)依据策略合并/替换初始化过滤器与运行期过滤器top_k、scale_score若未在run中传入则回退到构造函数中的值最终委托给document_store.bm25_retrieval(query..., filters..., top_k..., scale_score...)。bm25_retrieval是InMemoryDocumentStore的原生方法haystack/document_stores/in_memory/document_store.py异步版本为bm25_retrieval_async同文件第 1054 行附近由run_async调用。也就是说同步/异步只发生在 Document Store 层检索器本身是薄封装。InMemoryEmbeddingRetriever语义向量检索InMemoryEmbeddingRetriever检索与查询在语义上最相似的文档同样只搭配InMemoryDocumentStore。使用前必须保证文档与查询两侧都有 embedding 可用索引流水线中用DocumentEmbedder为文档生成向量查询流水线中用TextEmbedder为查询生成向量再传给检索器。完整用法示例Sentence Transformers 版API 文档给出的完整示例使用SentenceTransformersDocumentEmbedder/SentenceTransformersTextEmbedderfrom haystack import Document from haystack.components.embedders import SentenceTransformersDocumentEmbedder, SentenceTransformersTextEmbedder from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever from haystack.document_stores.in_memory import InMemoryDocumentStore docs [ Document(contentPython is a popular programming language), Document(contentpython ist eine beliebte Programmiersprache), ] doc_embedder SentenceTransformersDocumentEmbedder() doc_embedder.warm_up() docs_with_embeddings doc_embedder.run(docs)[documents] doc_store InMemoryDocumentStore() doc_store.write_documents(docs_with_embeddings) retriever InMemoryEmbeddingRetriever(doc_store) queryProgrammiersprache text_embedder SentenceTransformersTextEmbedder() text_embedder.warm_up() query_embedding text_embedder.run(query)[embedding] result retriever.run(query_embeddingquery_embedding) print(result[documents])注意与 BM25 的关键差异run的入参是query_embedding向量列表而非query字符串向量化完全由外部 Embedder 负责。构造参数与 run 参数构造函数in_memory/embedding_retriever.pydef __init__(document_store: InMemoryDocumentStore, filters: Optional[dict[str, Any]] None, top_k: int 10, scale_score: bool False, return_embedding: bool False, filter_policy: FilterPolicy FilterPolicy.REPLACE)参数默认值语义return_embeddingFalse为True时检索结果文档中附带各自的 embedding为False时仅返回文档本体减少内存占用与传输开销run签名component.output_types(documentslist[Document]) def run(query_embedding: list[float], filters: Optional[dict[str, Any]] None, top_k: Optional[int] None, scale_score: Optional[bool] None, return_embedding: Optional[bool] None)run_async签名与之完全相同。底层委托链源码第 166-182 行为apply_filter_policy(...)→ 参数回退 →document_store.embedding_retrieval(query_embedding..., filters..., top_k..., scale_score..., return_embedding...)异步版本调用embedding_retrieval_asyncdocument_store.py。构造函数与 BM25 版一致地对document_store做类型检查、对top_k 0抛ValueError。FilterPolicy 详解REPLACE 与 MERGEFilterPolicy是 Haystack 检索器统一的过滤策略枚举定义于 haystack/document_stores/types/filter_policy.pyREPLACE默认运行期传入的filters直接覆盖初始化时的过滤器。适用于需要针对不同查询动态切换过滤条件的场景。MERGE运行期过滤器与初始化过滤器合并重叠字段以运行期值为准从而进一步收窄搜索范围。合并逻辑由apply_filter_policy(filter_policy, init_filters, runtime_filters, default_logical_operator)实现同文件第 287 行起在MERGE模式下若两侧都是比较型过滤器则用combine_two_comparison_filters组合为逻辑表达式否则按默认逻辑操作符合并。由于filter_policy在to_dict中被序列化为字符串值、在from_dict中通过FilterPolicy.from_str还原见两个 InMemory 检索器的from_dict因此该配置可以完整地在 YAML/JSON 管道描述中往返。FilterRetriever纯元数据过滤检索FilterRetriever不计算任何相似度只按过滤器条件从 Document Store 中取出匹配的文档。它接受通用的DocumentStore不仅限 InMemory是搭建先过滤、再检索流水线的轻量工具。用法示例与运行期覆盖API 文档示例from haystack import Document from haystack.components.retrievers import FilterRetriever from haystack.document_stores.in_memory import InMemoryDocumentStore docs [ Document(contentPython is a popular programming language, meta{lang: en}), Document(contentpython ist eine beliebte Programmiersprache, meta{lang: de}), ] doc_store InMemoryDocumentStore() doc_store.write_documents(docs) retriever FilterRetriever(doc_store, filters{field: lang, operator: , value: en}) # 若在 run 中传入 filters将覆盖初始化时的过滤器 result retriever.run(filters{field: lang, operator: , value: de}) print(result[documents])__init__(document_store, filtersNone)与run(filtersNone)的参数语义完全一致从源码filter_retriever.py看run中的解析规则是resolved_filters filters if filters is not None else self.filters——运行期过滤器一旦给出就整体覆盖初始化值与FilterPolicy.REPLACE行为一致但该组件自身不携带 FilterPolicy 枚举。异步版run_async委托filter_documents_async同步版委托filter_documents。过滤器语法Comparison 与 Logic过滤器是嵌套字典支持两种类型规范见 haystack/document_stores/types/protocol.py比较型Comparison必须含field、operator、value三个键operator取值、!、、、、、in、not in。逻辑型Logic必须含operator与conditionsoperator取值NOT、OR、ANDconditions为比较型或逻辑型字典的列表。简单示例{field: meta.type, operator: , value: article}。复合示例filters { operator: AND, conditions: [ {field: meta.type, operator: , value: article}, {field: meta.date, operator: , value: 1420066800}, {field: meta.date, operator: , value: 1609455600}, {field: meta.rating, operator: , value: 3}, { operator: OR, conditions: [ {field: meta.genre, operator: in, value: [economy, politics]}, {field: meta.publisher, operator: , value: nytimes}, ], }, ], }SentenceWindowRetriever句子窗口上下文扩展SentenceWindowRetriever从 Document Store 中取回检索命中文档的邻近分块为查询结果补充上下文。它设计为接在某个基础 Retriever如 BM25、Embedding Retriever之后使用输出context_windows合并后的上下文文本列表与context_documents含邻近块的文档列表两类结果。前置条件分块元数据约定使用该组件的前提是文档携带两块元数据source_id标识分块所属的原始文档用于将同一来源的句子分块归组split_id分块在原始文档中的位置/顺序。这两个字段名可分别通过source_id_meta_field与split_id_meta_field自定义。与 DocumentSplitter 组合的端到端示例API 文档给出完整流水线示例DocumentSplitter生成的source_id/split_id/split_idx_start元数据正好满足约定from haystack import Document, Pipeline from haystack.components.retrievers.in_memory import InMemoryBM25Retriever from haystack.components.retrievers import SentenceWindowRetriever from haystack.components.preprocessors import DocumentSplitter from haystack.document_stores.in_memory import InMemoryDocumentStore splitter DocumentSplitter(split_length10, split_overlap5, split_byword) text ( This is a text with some words. There is a second sentence. And there is also a third sentence. It also contains a fourth sentence. And a fifth sentence. And a sixth sentence. And a seventh sentence ) doc Document(contenttext) docs splitter.run([doc]) doc_store InMemoryDocumentStore() doc_store.write_documents(docs[documents]) rag Pipeline() rag.add_component(bm25_retriever, InMemoryBM25Retriever(doc_store, top_k1)) rag.add_component(sentence_window_retriever, SentenceWindowRetriever(document_storedoc_store, window_size2)) rag.connect(bm25_retriever, sentence_window_retriever) rag.run({bm25_retriever: {query:third}})运行后sentence_window_retriever的输出形如{sentence_window_retriever: {context_windows: [some words. There is a second sentence. And there is also a third sentence. It also contains a fourth sentence. And a fifth sentence. And a sixth sentence. And a], context_documents: [Document(id..., content: some words. There is a second sentence. And there is , meta: {source_id: ..., page_number: 1, split_id: 1, split_idx_start: 20, ...}), Document(id..., content: second sentence. And there is also a third sentence. It , meta: {source_id: ..., split_id: 2, split_idx_start: 43, ...}), ...]}}命中third的分块split_id3位于窗口中心window_size2使其前后各取 2 个分块最终context_documents含 5 个文档并按split_idx_start排序。构造参数与 run 参数构造函数sentence_window_retriever.pydef __init__(document_store: DocumentStore, window_size: int 3, *, source_id_meta_field: Union[str, list[str]] source_id, split_id_meta_field: str split_id, raise_on_missing_meta_fields: bool True)参数默认值语义window_size3命中文档前后各取多少个邻近文档如2表示取前 2 个、后 2 个必须 0_validate_window_size会对 0抛ValueError源码第 247-250 行source_id_meta_fieldsource_id源 ID 元数据字段可传字符串或列表传列表时多个字段须全部匹配才视为同一来源split_id_meta_fieldsplit_id分块位置元数据字段raise_on_missing_meta_fieldsTrue为True时缺少必要元数据即抛ValueError为False时跳过缺失文档的上下文检索但仍把原文档放进结果并在日志中告警run签名component.output_types(context_windowslist[str], context_documentslist[Document]) def run(retrieved_documents: list[Document], window_size: Optional[int] None)运行期传入的window_size会覆盖构造时的值源码第 200 行。输出字典两个键context_windows与retrieved_documents一一对应的合并上下文文本每个字符串即一个上下文窗口context_documents原检索文档 其上下文文档按split_idx_start排序。底层实现过滤条件构造与去重叠合并从源码看_retrieve_context_for_document第 265-285 行每个命中文档的上下文获取分三步读取source_id可多字段与split_id任一缺失则跳过上下文检索并告警返回(doc.content or , [doc])。用_build_filter_conditions(split_id, window_size, source_ids)构造 AND 过滤条件第 310-322 行min_before split_id - window_size max_after split_id window_size conditions [ {field: fmeta.{split_id_meta_field}, operator: , value: min_before}, {field: fmeta.{split_id_meta_field}, operator: , value: max_after}, *source_id_filters, # 每个源字段一个 {field: fmeta.{...}, operator: , value: source_id} ] return {operator: AND, conditions: conditions}调用document_store.filter_documents(filter_conditions)取回候选再由静态方法merge_documents_text合并文本。merge_documents_text静态方法第 119-151 行的处理细节值得注意若文档元数据中都没有split_idx_start则直接拼接所有文本否则按split_idx_start排序后利用max(start, last_idx_end)跳过重叠区间实现去重叠合并——这正是DocumentSplitter(split_overlap0)场景下窗口文本不重复的关键。run_async走完全等价的异步路径_retrieve_context_for_document_async。兼容的 Document Store按 API 文档声明SentenceWindowRetriever与以下 Document Store 兼容Astra、Elasticsearch、OpenSearch、Pgvector、Pinecone、Qdrant这些 Store 均需实现filter_documents/filter_documents_async。AutoMergingRetriever父子层级文档自动合并AutoMergingRetriever基于阈值设定将命中的叶子节点文档替换为其父文档返回。它假设文档库中存在层级树状结构叶子节点被索引进 Document Store父节点保留层级关系。设计动机其设计动机非常直观一个段落被切成多个叶子分块后若某个查询命中了同一父节点下的多个分块那么整段父文档可能比单独的几个分块更有信息量。此时返回父文档能显著提升下游 LLM 的上下文质量。构造参数与阈值校验def __init__(document_store: DocumentStore, threshold: float 0.5)document_store从中检索父文档的 Document Storethreshold决定返回父文档还是保留子文档的阈值。必须满足0 threshold 1否则抛ValueError源码 auto_merging_retriever.py。合并判定公式对同一父节点的命中子文档数c与该父节点的全部子文档数n当c / n threshold时合并为父文档。例如父节点有 3 个子文档、命中其中 2 个时2/3 ≈ 0.67 0.5于是返回父文档。完整示例与 HierarchicalDocumentSplitter 搭配from haystack import Document from haystack.components.preprocessors import HierarchicalDocumentSplitter from haystack.components.retrievers.auto_merging_retriever import AutoMergingRetriever from haystack.document_stores.in_memory import InMemoryDocumentStore # 创建 3 层层级文档结构父文档有 3 个子文档 text The sun rose early in the morning. It cast a warm glow over the trees. Birds began to sing. original_document Document(contenttext) builder HierarchicalDocumentSplitter(block_sizes[10, 3], split_overlap0, split_byword) docs builder.run([original_document])[documents] # 存储 level-1 父文档并初始化检索器 doc_store_parents InMemoryDocumentStore() for doc in docs[documents]: if doc.meta[children_ids] and doc.meta[level] 1: doc_store_parents.write_documents([doc]) retriever AutoMergingRetriever(doc_store_parents, threshold0.5) # 假设检索到同一父节点下的 2 个叶子文档 # 父节点有 3 个子文档命中 2 个2/3 0.66(6)超过阈值 0.5应返回父文档 leaf_docs [doc for doc in docs[documents] if not doc.meta[children_ids]] docs retriever.run(leaf_docs[4:6])预期输出返回父文档而非两个叶子分块{documents: [Document(id538..), content: warm glow over the trees. Birds began to sing., meta: {block_size: 10, parent_id: 835.., children_ids: [c17..., 3ff..., 352...], level: 1, source_id: 835..., page_number: 1, split_id: 1, split_idx_start: 45})]}层级元数据约定与输入校验该组件依赖HierarchicalDocumentSplitter写入的元数据字段见 haystack/components/preprocessors/hierarchical_document_splitter.py__block_size当前节点块的尺寸根节点为 0__parent_id父文档 ID根节点为None__children_ids子文档 ID 列表叶子为空列表__level层级号根节点为 0逐层 1。run之前的_check_valid_documents源码第 101-113 行会校验每个输入文档必须含__parent_id、__level、__block_size三个字段缺失即抛ValueError。注意__level与__block_size使用存在性检查in doc.meta而非真值检查因为根节点的__level0、__block_size0本身是假值。递归合并算法run以及等价的run_async的核心是递归函数_try_merge_level源码第 141-167 行按__parent_id用defaultdict(list)对输入文档分组没有父节点的文档直接进入返回列表。对每个父组通过filter_documents({field: id, operator: , value: parent_id})从 Document Store 取回父文档要求恰好命中 1 个且父文档必须有子节点否则抛ValueError。计算score len(child_docs) / len(parent_doc.meta[__children_ids])若score threshold则合并为父文档否则子文档原样保留。若本轮产生了新的合并结果则递归地尝试向更高一层合并因为父文档本身也有__parent_id直到没有任何合并发生才返回。因此输出可能是不同层级文档的混合列表。兼容的 Document Store按 API 文档声明AutoMergingRetriever目前仅支持AstraDB、ElasticSearch、OpenSearch、PGVector、Qdrant这些 Store 需支持按 ID 过滤父文档。序列化to_dict / from_dict 契约上述五个组件全部实现统一的序列化契约to_dict() - dict[str, Any]将组件序列化为字典。InMemory 两个检索器还会把filter_policy序列化为字符串值filter_policy.value便于 YAML/JSON 表示。from_dict(cls, data) - 组件实例类方法反序列化。InMemory 两个检索器在from_dict中用FilterPolicy.from_str将字符串还原为枚举后再走default_from_dict。这使得整个检索组件可以被完整嵌入 Haystack 的 YAML 管道描述见 haystack/marshal/yaml.py与Pipeline.loads/Pipeline.dumps流程实现声明式定义、可复现部署。选型速查五个检索器怎么选场景推荐组件关键理由关键词/词法匹配无需模型InMemoryBM25Retriever纯 BM25 算法无需 embedding速度快、零模型依赖语义相似检索已具备向量InMemoryEmbeddingRetriever接收query_embedding配合TextEmbedder/DocumentEmbedder只按元数据条件过滤FilterRetriever不做相似度计算仅执行过滤器Comparison/Logic 语法检索后需要上下文扩展SentenceWindowRetriever基于source_id/split_id拉取邻近分块并去重叠合并层级分块下避免碎片化召回AutoMergingRetriever命中同一父节点分块比例超阈值时自动升级为父文档结语Haystack 2.x 的 Retrievers API 以组件化、可序列化、可异步为设计主线BM25 与 Embedding 检索器通过run/run_async双入口薄封装 Document Store 的原生检索方法FilterPolicy统一了运行期与初始化过滤器的 REPLACE/MERGE 语义SentenceWindowRetriever与AutoMergingRetriever则分别解决了碎片化上下文与碎片化召回两类实际问题。理解这些组件的参数语义与底层调用链你就能依据业务场景快速搭建出高质量的 RAG 检索阶段。【免费下载链接】haystackOpen-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.项目地址: https://gitcode.com/GitHub_Trending/ha/haystack创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考