微信扫码
添加专属顾问
搜索和重新排名:提高结果相关性
搜索系统通常采用两种主要方法:关键字和语义。关键字搜索将精确的查询词与索引数据库内容匹配,而语义搜索使用 NLP 和机器学习来理解查询上下文和意图。许多有效的系统结合了这两种方法以获得最佳结果。
初始检索后,重新排序可以进一步提高结果相关性。传统的重新排序依赖于历史用户交互数据,但这种方法难以处理新内容,并且需要大量数据才能有效训练。一种先进的替代方法是使用交叉编码器,它直接比较查询结果对的相似性。
交叉编码器直接比较两段文本并计算相似度得分。与传统的语义搜索方法不同,我们无法预先计算交叉编码器的嵌入并在以后重复使用它们。相反,我们必须对每一对想要比较的文本运行交叉编码器,这使得这种方法在计算上非常昂贵,并且不适用于大规模搜索。但是,它对于重新排序我们数据集的子集非常有效,因为它擅长评估新的、未见过的数据,而无需大量用户交互数据进行微调。
交叉编码器弥补了传统重排序系统在深度文本分析方面的局限性,尤其是针对新颖或高度特定内容。它们不依赖大量用户交互数据集进行训练(尽管此类数据仍然很有用),并且擅长处理新数据和以前从未见过的数据。这使得交叉编码器成为在重排序环境中增强搜索结果相关性的绝佳选择。
实施重新排名
我们将使用 LlamaIndex 和 PostgresML 托管索引实现一个简单的重新排名示例。有关 PostgresML 托管索引的更多信息。查看我们关于 LlamaIndex 的公告:使用 LlamaIndex + PostgresML 简化您的 RAG 应用程序架构 。
安装所需的依赖项以开始使用:
pip install llama_index llama-index-indices-managed-postgresml
我们将使用 Paul Graham 数据集,可以通过 curl 下载:
mkdir data
curl -o data/paul_graham_essay.txt https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt
PostgresML 托管索引将处理存储、拆分、嵌入和查询我们的文档。我们需要的只是一个数据库连接字符串。如果您还没有, 请创建您的 PostgresML 帐户 。完成您的个人资料后,您将获得 100 美元的免费积分。
设置 PGML_DATABASE_URL 环境变量:
export PGML_DATABASE_URL="{YOUR_CONNCECTION_STRING}"
让我们创建索引:
from llama_index.core.readers import SimpleDirectoryReader
from llama_index.indices.managed.postgresml import PostgresMLIndex
documents = SimpleDirectoryReader("data").load_data()
index = PostgresMLIndex.from_documents(
documents, collection_name="llama-index-rerank-example"
)
请注意,collection_name 用于唯一标识您正在使用的索引。
这里我们使用 SimpleDirectoryReader 来加载文档,然后从这些文档构造 PostgresMLIndex。
此工作流程不需要文档预处理。相反,文档会直接发送到 PostgresML,并根据管道规范进行存储、拆分和嵌入。这是使用 PostgresML 托管索引的独特品质。
现在让我们搜索一下!我们可以执行语义搜索,并通过从索引中创建检索器来获取前 2 个结果。
retriever = index.as_retriever(limit=2)
docs = retriever.retrieve("What did the author do as a child?")
for doc in docs:
print("---------")
print(f"Id: {doc.id_}")
print(f"Score: {doc.score}")
print(f"Text: {doc.text}")
这样做我们得到:
---------
Id: de01b7e1-95f8-4aa0-b4ec-45ef64816e0e
Score: 0.7793415653313153
Text: Wow, I thought, there's an audience. If I write something and put it on the web, anyone can read it. That may seem obvious now, but it was surprising then. In the print era there was a narrow channel to readers, guarded by fierce monsters known as editors. The only way to get an audience for anything you wrote was to get it published as a book, or in a newspaper or magazine. Now anyone could publish anything.
This had been possible in principle since 1993, but not many people had realized it yet. I had been intimately involved with building the infrastructure of the web for most of that time, and a writer as well, and it had taken me 8 years to realize it. Even then it took me several years to understand the implications. It meant there would be a whole new generation of essays. [11]
In the print era, the channel for publishing essays had been vanishingly small. Except for a few officially anointed thinkers who went to the right parties in New York, the only people allowed to publish essays were specialists writing about their specialties. There were so many essays that had never been written, because there had been no way to publish them. Now they could be, and I was going to write them. [12]
I've worked on several different things, but to the extent there was a turning point where I figured out what to work on, it was when I started publishing essays online. From then on I knew that whatever else I did, I'd always write essays too.
---------
Id: de01b7e1-95f8-4aa0-b4ec-45ef64816e0e
Score: 0.7770352826735559
Text: Asterix comics begin by zooming in on a tiny corner of Roman Gaul that turns out not to be controlled by the Romans. You can do something similar on a map of New York City: if you zoom in on the Upper East Side, there's a tiny corner that's not rich, or at least wasn't in 1993. It's called Yorkville, and that was my new home. Now I was a New York artist — in the strictly technical sense of making paintings and living in New York.
I was nervous about money, because I could sense that Interleaf was on the way down. Freelance Lisp hacking work was very rare, and I didn't want to have to program in another language, which in those days would have meant C++ if I was lucky. So with my unerring nose for financial opportunity, I decided to write another book on Lisp. This would be a popular book, the sort of book that could be used as a textbook. I imagined myself living frugally off the royalties and spending all my time painting. (The painting on the cover of this book, ANSI Common Lisp, is one that I painted around this time.)
The best thing about New York for me was the presence of Idelle and Julian Weber. Idelle Weber was a painter, one of the early photorealists, and I'd taken her painting class at Harvard. I've never known a teacher more beloved by her students. Large numbers of former students kept in touch with her, including me. After I moved to New York I became her de facto studio assistant.
这些结果还不错,但并不完美。让我们尝试使用交叉编码器进行重新排序。
retriever = index.as_retriever(
limit=2,
rerank={
"model": "mixedbread-ai/mxbai-rerank-base-v1",
"num_documents_to_rerank": 100
}
)
docs = retriever.retrieve("What did the author do as a child?")
for doc in docs:
print("---------")
print(f"Id: {doc.id_}")
print(f"Score: {doc.score}")
print(f"Text: {doc.text}")
在这里,我们将检索器配置为返回排名前两个的文档,但这次,我们添加了一个重新排名参数以使用mixedbread-ai/mxbai-rerank-base-v1模型。这意味着我们的初始语义搜索将返回 100 个结果,然后由mixedbread-ai/mxbai-rerank-base-v1模型对这些结果进行重新排名,并且仅显示排名前两个的结果。
运行此输出:
Id: de01b7e1-95f8-4aa0-b4ec-45ef64816e0e
Score: 0.17803585529327393
Text: What I Worked On
February 2021
Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.
The first programs I tried writing were on the IBM 1401 that our school district used for what was then called "data processing." This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.
The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.
---------
Id: de01b7e1-95f8-4aa0-b4ec-45ef64816e0e
Score: 0.1057136133313179
Text: I wanted not just to build things, but to build things that would last.
In this dissatisfied state I went in 1988 to visit Rich Draves at CMU, where he was in grad school. One day I went to visit the Carnegie Institute, where I'd spent a lot of time as a kid. While looking at a painting there I realized something that might seem obvious, but was a big surprise to me. There, right on the wall, was something you could make that would last. Paintings didn't become obsolete. Some of the best ones were hundreds of years old.
And moreover this was something you could make a living doing. Not as easily as you could by writing software, of course, but I thought if you were really industrious and lived really cheaply, it had to be possible to make enough to survive. And as an artist you could be truly independent. You wouldn't have a boss, or even need to get research funding.
I had always liked looking at paintings. Could I make them? I had no idea. I'd never imagined it was even possible. I knew intellectually that people made art — that it didn't just appear spontaneously — but it was as if the people who made it were a different species. They either lived long ago or were mysterious geniuses doing strange things in profiles in Life magazine. The idea of actually being able to make art, to put that verb before that noun, seemed almost miraculous.
这些结果好多了!我们可以看到,最上面的文档回答了用户的问题。请注意,我们不必指定用于重新排名的第三方 API。再一次,PostgresML 使用数据库中的交叉编码器来处理重新排名。
我们可以在 RAG 中直接使用重新排序:
query_engine = index.as_query_engine(
streaming=True,
vector_search_limit=2,
vector_search_rerank={
"model": "mixedbread-ai/mxbai-rerank-base-v1",
"num_documents_to_rerank": 100,
},
)
results = query_engine.query("What did the author do as a child?")
for text in results.response_gen:
print(text, end="", flush=True)
运行此输出:
Based on the context information, as a child, the author worked on writing (writing short stories) and programming (on the IBM 1401 using Fortran) outside of school.
这正是我们想要的答案!
重新排序可带来更好的结果
搜索可能很复杂。使用交叉编码器进行重新排序可以通过比较文本对并有效处理新数据来改进搜索。使用 LlamaIndex 和 PostgresML 实现重新排序可以改进搜索结果,在检索增强生成应用程序中提供更精确的答案。
要开始使用 PostgresML 和 LlamaIndex,您可以按照 PostgresML 入门 指南 设置您的帐户,并将上述示例与您自己的数据一起使用。
53AI,企业落地大模型首选服务商
产品:场景落地咨询+大模型应用平台+行业解决方案
承诺:免费POC验证,效果达标后再合作。零风险落地应用大模型,已交付160+中大型企业
2026-07-01
提升 RAG 准确率全攻略 让你的 AI 知识库 真正靠谱起来!
2026-06-30
教程:如何用AutoRAG + Milvus避免RAG 与Agent 中出现串租问题
2026-06-30
知识库不是文件堆——我把RAG准确率从60%调到了92%
2026-06-30
本体论语义建设新思路,另类RAG来解决检索问题
2026-06-30
别把RAG当架构:Ontology(本体)才是Agent的业务世界
2026-06-29
PixelRAG:伯克利团队颠覆传统 RAG,用截图代替文本检索! 28 天狂揽 3000+ Star!
2026-06-29
腾讯WeKnora开源详解(三):检索引擎与生态集成
2026-06-29
腾讯开源WeKnora详解(二):知识库与对话核心能力
2026-04-06
2026-04-27
2026-04-23
2026-04-20
2026-04-09
2026-04-12
2026-04-22
2026-04-10
2026-05-14
2026-04-30
2026-06-23
2026-06-23
2026-06-15
2026-06-10
2026-06-10
2026-05-20
2026-05-18
2026-05-11
欢迎您使用【53AI 官方网站】(以下简称“本网站”或“我们”)。本《会员服务协议》(以下简称“本协议”)是您(以下简称“会员”或“用户”)与【深圳市博思协创网络科技有限公司】之间关于注册、登录及使用本网站会员服务所订立的法律协议。
在您注册或登录前,请务必审慎阅读、充分理解各条款内容,特别是免除或限制责任的条款、知识产权条款、争议解决条款等。此类条款将以加粗形式提示您注意。 当您通过微信公众号授权、手机验证码验证或其他方式成功登录本网站时,即视为您已完全理解并同意接受本协议的全部内容。
一、 定义
本网站:指由【深圳市博思协创网络科技有限公司】运营的,域名为【53ai.com】的网站及相关移动端页面。
会员服务:指本网站向注册会员提供的知识库文章查阅、内容检索及其他相关增值服务。
知识库内容:指本网站发布的包括但不限于文字、图表、数据、研究报告、行业分析等数字化内容资源。
二、 账号注册与登录
登录方式:本网站支持以下登录方式,您可根据实际情况选择:
微信公众号授权登录:您同意将您的微信OpenID信息授权给本网站,用于创建或关联会员账号。
手机验证码登录:您需提供真实有效的手机号码,并通过短信验证码完成身份验证与登录/注册。
账号安全:您的账号仅限您本人使用,禁止赠与、借用、租用、转让或售卖。因您保管不善导致的账号被盗、密码泄露等损失,由您自行承担。
实名认证:根据相关法律法规要求,我们可能要求您在特定功能下完成实名认证。如您拒绝提供,可能无法使用部分或全部服务。
未成年人保护:若您未满18周岁,请在法定监护人的陪同下阅读本协议,并在征得监护人同意后使用本服务。
三、 服务内容与规范
知识库查阅权限:会员登录后,有权按照其会员等级对应的权限范围,在线浏览、检索本网站知识库中的相关文章及内容。
服务变更:我们有权根据业务发展需要,调整、变更或终止部分服务内容,并将以网站公告、公众号消息等方式提前通知。
禁止行为:您在使用服务时不得实施以下行为:
利用技术手段批量爬取、下载、转存知识库内容;
将知识库内容用于商业目的或未经授权地向第三方传播;
干扰本网站正常运行或侵犯其他用户合法权益;
发布违法违规信息或从事违反公序良俗的活动。
四、 知识产权声明
权利归属:本网站知识库中的排版设计、软件代码等内容的知识产权均归【公司全称】或原权利人所有,受《中华人民共和国著作权法》等法律保护。
有限许可:本网站授予会员一项非独占、不可转让、不可转授权的普通许可,仅限于个人学习、研究之目的在线查阅知识库内容。
侵权追责:未经书面许可,任何单位或个人不得以任何形式复制、转载、摘编、镜像、汇编或以其他方式使用上述内容。一经发现,我们保留追究其法律责任的权利。
五、 个人信息保护
我们重视对您个人信息的保护。关于我们如何收集、使用、存储和保护您的个人信息,请单独阅读 《隐私政策》。
您通过微信公众号授权或手机号验证所提供的信息,我们将严格按照《个人信息保护法》的规定处理,仅用于身份识别、服务提供及安全验证等必要用途。
您可以随时通过网站设置或联系客服行使查阅、更正、删除个人信息及撤回授权同意的权利。
六、 免责声明
内容准确性:知识库内容仅供参考,不构成专业建议。我们不对其完整性、准确性、时效性作任何明示或暗示的保证,您应自行判断并承担使用风险。
不可抗力:因自然灾害、政策法规变化、网络故障、第三方平台接口异常(如微信接口维护、运营商短信通道故障)等不可抗力导致的服务中断或延迟,我们不承担违约责任。
第三方链接:本网站可能包含指向第三方网站的链接,该等网站的内容和服务不受我们控制,请您自行甄别风险。
七、 违约责任
如您违反本协议约定,我们有权视情节采取警告、限制功能、暂停服务、注销账号等措施,并保留要求赔偿损失的权利。
如因您的违约行为导致我们遭受行政处罚、第三方索赔或商誉损失,您应承担全部赔偿责任(包括但不限于罚款、赔偿金、律师费、公证费等)。
八、 法律适用与争议解决
本协议的订立、执行和解释均适用中华人民共和国大陆地区法律。
因本协议产生的或与本协议有关的任何争议,双方应友好协商解决;协商不成的,任何一方均可向【公司所在地】有管辖权的人民法院提起诉讼。
九、 其他
本协议构成双方就本服务达成的完整协议,取代此前任何口头或书面约定。
本协议任一条款被认定为无效或不可执行的,不影响其他条款的效力。
我们对本协议享有最终解释权,并在法律允许的范围内保留随时修改的权利。修改后的协议一经公布即生效,继续使用服务即视为同意修订内容。