构建基于LangChain的语义搜索引擎

import EmbeddingsTabsPy from '/snippets/embeddings-tabs-py.mdx';
import EmbeddingsTabsJS from '/snippets/embeddings-tabs-js.mdx';
import VectorstoreTabsPy from '/snippets/vectorstore-tabs-py.mdx';
import VectorstoreTabsJS from '/snippets/vectorstore-tabs-js.mdx';

概述

本教程将使您熟悉LangChain的文档加载器、嵌入和向量存储抽象。这些抽象旨在支持从（向量）数据库和其他来源检索数据，以便与LLM工作流程集成。对于需要检索数据作为模型推理一部分的应用程序，例如检索增强生成（RAG）或检索增强生成，它们非常重要。

在这里，我们将构建一个在PDF文档上的搜索引擎。这将使我们能够检索与输入查询相似的PDF中的段落。指南还包括在搜索引擎之上的最小化RAG实现。

概念

本指南专注于文本数据的检索。我们将涵盖以下概念：

• 文档和文档加载器;
• 文本分割器;
• 嵌入向量;
• 向量存储 和 检索器.

安装

安装

本教程需要 langchain-community 和 pypdf 包：

pip

pip install langchain-community pypdf

有关详细信息，请参阅我们的安装指南。

LangSmith

许多您使用LangChain构建的应用程序将包含多个步骤，以及多次调用LLM调用。 随着这些应用程序变得越来越复杂，能够检查您的链或智能体内部究竟发生了什么变得至关重要。 要做到这一点，最佳方式是使用LangSmith。

在您通过上述链接注册后，请确保设置环境变量以开始记录跟踪信息：

export LANGSMITH_TRACING="true"
export LANGSMITH_API_KEY="..."

或者，如果在笔记本中，您可以使用以下方式设置它们：

import getpass
import os

os.environ["LANGSMITH_TRACING"] = "true"
os.environ["LANGSMITH_API_KEY"] = getpass.getpass()

1. 文档和文档加载器

LangChain 实现了一个 文档 抽象，旨在表示一个文本单元及其相关元数据。它有三个属性：

• page_content：表示内容的字符串；
• metadata：包含任意元数据的字典；
• id：（可选）文档的字符串标识符。

metadata 属性可以捕获有关文档来源、与其他文档的关系以及其他信息。请注意，一个单独的 Document 对象通常代表一个较大文档的一部分。

我们可以根据需要生成示例文档：

from langchain_core.documents import Document

documents = [
    Document(
        page_content="Dogs are great companions, known for their loyalty and friendliness.",
        metadata={"source": "mammal-pets-doc"},
    ),
    Document(
        page_content="Cats are independent pets that often enjoy their own space.",
        metadata={"source": "mammal-pets-doc"},
    ),
]

然而，LangChain生态系统实现了文档加载器，这些加载器与数百个常见来源集成。这使得将来自这些来源的数据整合到您的AI应用中变得容易。

加载文档

让我们将一个PDF文件加载成一系列Document对象。这是一个示例PDF——耐克2023年的10-K文件。我们可以查阅LangChain文档了解可用的PDF文档加载器。

from langchain_community.document_loaders import PyPDFLoader

file_path = "../example_data/nke-10k-2023.pdf"
loader = PyPDFLoader(file_path)

docs = loader.load()

print(len(docs))

107

PyPDFLoader 在每页PDF中加载一个 Document 对象。对于每个对象，我们可以轻松访问：

• 页面的字符串内容；
• 包含文件名和页码的元数据。

print(f"{docs[0].page_content[:200]}\n")
print(docs[0].metadata)

Table of Contents
UNITED STATES
SECURITIES AND EXCHANGE COMMISSION
Washington, D.C. 20549
FORM 10-K
(Mark One)
☑ ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934
FO

{'source': '../example_data/nke-10k-2023.pdf', 'page': 0}

分割

为了信息检索和下游问答的目的，一个页面可能是一个过于粗略的表示。我们的最终目标将是检索Document对象，这些对象能够回答输入查询，进一步拆分我们的PDF将有助于确保文档相关部分的含义不会被周围文本“冲淡”。

我们可以使用 文本分割器 来实现这个目的。在这里，我们将使用一个简单的基于字符的文本分割器。我们将把我们的文档分割成1000个字符的块，块与块之间有200个字符的重叠。重叠有助于减少将一个句子与其相关的重要上下文分开的可能性。我们使用 RecursiveCharacterTextSplitter，它将递归地使用常见的分隔符（如换行符）来分割文档，直到每个块达到适当的大小。这是通用文本用例推荐的文本分割器。

我们将 add_start_index=True 设置为保留每个分割文档在初始文档中开始的字符索引，作为元数据属性“start_index”。

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000, chunk_overlap=200, add_start_index=True
)
all_splits = text_splitter.split_documents(docs)

print(len(all_splits))

514

2. 嵌入

向量搜索是存储和搜索非结构化数据（如非结构化文本）的一种常见方式。其基本思想是存储与文本相关联的数值向量。给定一个查询，我们可以将其嵌入为一个相同维度的向量，并使用向量相似度度量（如余弦相似度）来识别相关文本。

LangChain 支持来自 数十个提供商 的嵌入。这些模型指定了如何将文本转换为数值向量。让我们选择一个模型：

vector_1 = embeddings.embed_query(all_splits[0].page_content)
vector_2 = embeddings.embed_query(all_splits[1].page_content)

assert len(vector_1) == len(vector_2)
print(f"Generated vectors of length {len(vector_1)}\n")
print(vector_1[:10])

Generated vectors of length 1536

[-0.008586574345827103, -0.03341241180896759, -0.008936782367527485, -0.0036674530711025, 0.010564599186182022, 0.009598285891115665, -0.028587326407432556, -0.015824200585484505, 0.0030416189692914486, -0.012899317778646946]

拥有一个用于生成文本嵌入的模型后，我们接下来可以将它们存储在一个支持高效相似性搜索的特殊数据结构中。

3. 向量存储

LangChain 向量存储对象包含向存储中添加文本和Document对象的方法，并使用各种相似度指标查询它们。它们通常使用嵌入模型初始化，这些模型决定了文本数据如何转换为数值向量。

LangChain 包含一套 集成，支持不同的向量存储技术。一些向量存储由提供商托管（例如，各种云提供商），并需要特定的凭证才能使用；一些（如 Postgres) 在独立的基础设施中运行，可以本地运行或通过第三方运行；其他可以在内存中运行，适用于轻量级工作负载。让我们选择一个向量存储：

实例化我们的向量存储后，我们现在可以索引文档了。

ids = vector_store.add_documents(documents=all_splits)

请注意，大多数向量存储实现都允许您连接到现有的向量存储——例如，通过提供客户端、索引名称或其他信息。有关特定集成的更多详细信息，请参阅文档。

一旦实例化了一个包含文档的 VectorStore，我们就可以对其进行查询。VectorStore 包含以下查询方法：

• 同步和异步；
• 通过字符串查询和通过向量；
• 带有和没有返回相似度分数；
• 通过相似度和@[最大边际相关性][VectorStore.max_marginal_relevance_search]（以平衡相似度与查询的多样性，以在检索结果中实现多样性）。

方法通常会在其输出中包含一个文档对象的列表。

用法

嵌入通常将文本表示为“密集”向量，使得具有相似意义的文本在几何上是接近的。这使得我们只需输入一个问题，就可以检索相关信息，而无需了解文档中使用的任何特定关键词。

根据字符串查询的相似度返回文档：

results = vector_store.similarity_search(
    "How many distribution centers does Nike have in the US?"
)

print(results[0])

page_content='direct to consumer operations sell products through the following number of retail stores in the United States:
U.S. RETAIL STORES NUMBER
NIKE Brand factory stores 213
NIKE Brand in-line stores (including employee-only stores) 74
Converse stores (including factory stores) 82
TOTAL 369
In the United States, NIKE has eight significant distribution centers. Refer to Item 2. Properties for further information.
2023 FORM 10-K 2' metadata={'page': 4, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 3125}

异步查询

results = await vector_store.asimilarity_search("When was Nike incorporated?")

print(results[0])

page_content='Table of Contents
PART I
ITEM 1. BUSINESS
GENERAL
NIKE, Inc. was incorporated in 1967 under the laws of the State of Oregon. As used in this Annual Report on Form 10-K (this "Annual Report"), the terms "we," "us," "our,"
"NIKE" and the "Company" refer to NIKE, Inc. and its predecessors, subsidiaries and affiliates, collectively, unless the context indicates otherwise.
Our principal business activity is the design, development and worldwide marketing and selling of athletic footwear, apparel, equipment, accessories and services. NIKE is
the largest seller of athletic footwear and apparel in the world. We sell our products through NIKE Direct operations, which are comprised of both NIKE-owned retail stores
and sales through our digital platforms (also referred to as "NIKE Brand Digital"), to retail accounts and to a mix of independent distributors, licensees and sales' metadata={'page': 3, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}

返回分数：

# Note that providers implement different scores; the score here
# is a distance metric that varies inversely with similarity.

results = vector_store.similarity_search_with_score("What was Nike's revenue in 2023?")
doc, score = results[0]
print(f"Score: {score}\n")
print(doc)

Score: 0.23699893057346344

page_content='Table of Contents
FISCAL 2023 NIKE BRAND REVENUE HIGHLIGHTS
The following tables present NIKE Brand revenues disaggregated by reportable operating segment, distribution channel and major product line:
FISCAL 2023 COMPARED TO FISCAL 2022
•NIKE, Inc. Revenues were $51.2 billion in fiscal 2023, which increased 10% and 16% compared to fiscal 2022 on a reported and currency-neutral basis, respectively.
The increase was due to higher revenues in North America, Europe, Middle East & Africa ("EMEA"), APLA and Greater China, which contributed approximately 7, 6,
2 and 1 percentage points to NIKE, Inc. Revenues, respectively.
•NIKE Brand revenues, which represented over 90% of NIKE, Inc. Revenues, increased 10% and 16% on a reported and currency-neutral basis, respectively. This
increase was primarily due to higher revenues in Men's, the Jordan Brand, Women's and Kids' which grew 17%, 35%,11% and 10%, respectively, on a wholesale
equivalent basis.' metadata={'page': 35, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}

根据嵌入查询的相似度返回文档：

embedding = embeddings.embed_query("How were Nike's margins impacted in 2023?")

results = vector_store.similarity_search_by_vector(embedding)
print(results[0])

page_content='Table of Contents
GROSS MARGIN
FISCAL 2023 COMPARED TO FISCAL 2022
For fiscal 2023, our consolidated gross profit increased 4% to $22,292 million compared to $21,479 million for fiscal 2022. Gross margin decreased 250 basis points to
43.5% for fiscal 2023 compared to 46.0% for fiscal 2022 due to the following:
*Wholesale equivalent
The decrease in gross margin for fiscal 2023 was primarily due to:
•Higher NIKE Brand product costs, on a wholesale equivalent basis, primarily due to higher input costs and elevated inbound freight and logistics costs as well as
product mix;
•Lower margin in our NIKE Direct business, driven by higher promotional activity to liquidate inventory in the current period compared to lower promotional activity in
the prior period resulting from lower available inventory supply;
•Unfavorable changes in net foreign currency exchange rates, including hedges; and
•Lower off-price margin, on a wholesale equivalent basis.
This was partially offset by:' metadata={'page': 36, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}

了解更多：

• API参考
• 特定集成文档

4. 检索器

LangChain VectorStore 对象不继承自 @[Runnable]。LangChain @[Retrievers] 是可运行的，因此它们实现了一套标准方法（例如，同步和异步的 invoke 和 batch 操作）。尽管我们可以从向量存储中构建检索器，但检索器也可以与数据源的非向量存储接口（例如外部API）进行交互。

我们可以自己创建一个简单版本，无需继承 Retriever。如果我们选择想要用来检索文档的方法，我们可以轻松地创建一个可运行的版本。下面我们将围绕 similarity_search 方法构建一个：

from typing import List

from langchain_core.documents import Document
from langchain_core.runnables import chain


@chain
def retriever(query: str) -> List[Document]:
    return vector_store.similarity_search(query, k=1)


retriever.batch(
    [
        "How many distribution centers does Nike have in the US?",
        "When was Nike incorporated?",
    ],
)

[[Document(metadata={'page': 4, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 3125}, page_content='direct to consumer operations sell products through the following number of retail stores in the United States:\nU.S. RETAIL STORES NUMBER\nNIKE Brand factory stores 213 \nNIKE Brand in-line stores (including employee-only stores) 74 \nConverse stores (including factory stores) 82 \nTOTAL 369 \nIn the United States, NIKE has eight significant distribution centers. Refer to Item 2. Properties for further information.\n2023 FORM 10-K 2')],
 [Document(metadata={'page': 3, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}, page_content='Table of Contents\nPART I\nITEM 1. BUSINESS\nGENERAL\nNIKE, Inc. was incorporated in 1967 under the laws of the State of Oregon. As used in this Annual Report on Form 10-K (this "Annual Report"), the terms "we," "us," "our,"\n"NIKE" and the "Company" refer to NIKE, Inc. and its predecessors, subsidiaries and affiliates, collectively, unless the context indicates otherwise.\nOur principal business activity is the design, development and worldwide marketing and selling of athletic footwear, apparel, equipment, accessories and services. NIKE is\nthe largest seller of athletic footwear and apparel in the world. We sell our products through NIKE Direct operations, which are comprised of both NIKE-owned retail stores\nand sales through our digital platforms (also referred to as "NIKE Brand Digital"), to retail accounts and to a mix of independent distributors, licensees and sales')]]

向量存储实现了一个 as_retriever 方法，该方法将生成一个检索器，具体为 VectorStoreRetriever。这些检索器包括特定的 search_type 和 search_kwargs 属性，用于标识要调用底层向量存储的哪些方法，以及如何参数化它们。例如，我们可以用以下方式复制上述操作：

retriever = vector_store.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 1},
)

retriever.batch(
    [
        "How many distribution centers does Nike have in the US?",
        "When was Nike incorporated?",
    ],
)

[[Document(metadata={'page': 4, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 3125}, page_content='direct to consumer operations sell products through the following number of retail stores in the United States:\nU.S. RETAIL STORES NUMBER\nNIKE Brand factory stores 213 \nNIKE Brand in-line stores (including employee-only stores) 74 \nConverse stores (including factory stores) 82 \nTOTAL 369 \nIn the United States, NIKE has eight significant distribution centers. Refer to Item 2. Properties for further information.\n2023 FORM 10-K 2')],
 [Document(metadata={'page': 3, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}, page_content='Table of Contents\nPART I\nITEM 1. BUSINESS\nGENERAL\nNIKE, Inc. was incorporated in 1967 under the laws of the State of Oregon. As used in this Annual Report on Form 10-K (this "Annual Report"), the terms "we," "us," "our,"\n"NIKE" and the "Company" refer to NIKE, Inc. and its predecessors, subsidiaries and affiliates, collectively, unless the context indicates otherwise.\nOur principal business activity is the design, development and worldwide marketing and selling of athletic footwear, apparel, equipment, accessories and services. NIKE is\nthe largest seller of athletic footwear and apparel in the world. We sell our products through NIKE Direct operations, which are comprised of both NIKE-owned retail stores\nand sales through our digital platforms (also referred to as "NIKE Brand Digital"), to retail accounts and to a mix of independent distributors, licensees and sales')]]

VectorStoreRetriever 支持以下搜索类型："similarity"（默认）、"mmr"（最大边际相关性，如上所述）和"similarity_score_threshold"。我们可以使用后者通过相似度分数来阈值化检索器输出的文档。

智能体可以轻松集成到更复杂的应用中，例如将给定问题与检索到的上下文结合成一个用于LLM的提示的检索增强生成（RAG）应用。要了解更多关于构建此类应用的信息，请查看RAG教程。

下一步

您现在已经看到了如何在PDF文档上构建一个语义搜索引擎的方法。

关于文档加载器的更多信息：

• 概述
• 可用集成

关于嵌入体的更多信息：

• 概述
• 可用集成

关于向量存储的更多信息：

• 概述
• 可用集成

关于RAG的更多信息，请参阅：

• 构建检索增强生成 (RAG) 应用程序

---

在GitHub上编辑此页面的源代码。

通过MCP将这些文档编程连接到Claude、VSCode等，以获取实时答案。