上篇文章提到，我们将文本切分划分为五个层级，并介绍了第一个层级的实现和一些基础知识。今天我们来介绍第二个层级Recursive Character Text Splitting，这个切分器是日常实践中通常会采用的，是最直观也有效的方案，也是其它高级方案的基础。文本切分五个层级：
Level 1: Character Splitting - 简单的字符长度切分
Level 2: Recursive Character Text Splitting - 通过分隔符切分，然后递归合并
Level 3: Document Specific Splitting - 针对不同文档格式切分 (PDF, Python, Markdown)
Level 4: Semantic Splitting - 语义切分
Level 5: Agentic Splitting-使用代理实现自动切分
为什么需要Recursive Character Text Splitting
按字符长度切分的问题在于，根本没有考虑到文档的结构。只是按固定数量的字符分割，这样很容易造成文档被随意切分开了，导致最后的效果不好。
递归字符文本分割器可以帮助解决这个问题。有了它，切分的过程可描述为：我们先指定一系列分隔符，这些分隔符用于分割文档，然后将分割好的小块合并，达到指定的长度。
它是切分器中最基础的做法，也是快速搭建应用流程的首选。如果在不知道选择哪个切分器的时候，这是一个不错的选择。
我们看看在langchain中的默认分隔符["\n\n", "\n", " ", ""]：
看看下面例子，在一般情况下，我们希望的切分情况，如何能实现它？
text = """One of the most important things I didn't understand about the world when I was a child is the degree to which the returns for performance are superlinear.
Teachers and coaches implicitly told us the returns were linear. "You get out," I heard a thousand times, "what you put in." They meant well, but this is rarely true. If your product is only half as good as your competitor's, you don't get half as many customers. You get no customers, and you go out of business.
It's obviously true that the returns for performance are superlinear in business. Some think this is a flaw of capitalism, and that if we changed the rules it would stop being true. But superlinear returns for performance are a feature of the world, not an artifact of rules we've invented. We see the same pattern in fame, power, military victories, knowledge, and even benefit to humanity. In all of these, the rich get richer. [1]"""
可以很直观地将它切分成三个段落：
如何实现
加载依赖方法
from langchain.text_splitter import RecursiveCharacterTextSplitter
构建切分器
text_splitter = RecursiveCharacterTextSplitter(chunk_size=65, chunk_overlap=0)text_splitter.create_documents([text])# chunk_size 每个片段的字符数# chunk_overlap 片段之间的重叠字符数
总共生成了16段，从结果上看，虽然没有再将单词切开了，但感觉切的还是太零碎了：
注意，现在有很多块以“.”结尾。这是因为这些是句子的结尾，但又没有把“.”添加到分隔符中，所以比较多。我也不知道为什么“ ”是分隔符，“.”不是默认分隔符，有知道的可以私信我，哈哈。拆分器首先寻找\n\n(段落断行)，一旦段落被分割，它就会查看块的大小，如果一个块太大，它就会被下一个分隔符分割。如果该块仍然太大，那么它将移动到下一个，依此类推。对英文来说65个字符也太小了，我们放大chunk_size看看效果。text_splitter = RecursiveCharacterTextSplitter(chunk_size=450, chunk_overlap=0)text_splitter.create_documents([text])
总共生成3段，这就是我们预期的切分效果。
仅仅使用字符个数作为小块内的切分依据，似乎不是一个好的方案；相同语义下，中英文、数字等字符个数差别太大了。下篇文章我们将介绍如何优化小块内的切分和计数方案。
粤ICP备14082021号
免费POC，
零成本试错
