LangChain 非常强大的一点就是封装了非常多强大的工具可以直接使用。降低了使用者的学习成本。比如数据网页爬取。
在其官方文档-网页爬取中，也有非常好的示例。
应用场景
实践应用
需求说明
实现思路
对应源码
from langchain_openai import ChatOpenAIllm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")
def extract(content: str, schema: dict):from langchain.chains import create_extraction_chainreturn create_extraction_chain(schema=schema, llm=llm).invoke(content)
import pprintfrom langchain_text_splitters import RecursiveCharacterTextSplitterdef scrape_with_playwright(urls, schema):loader = AsyncChromiumLoader(urls)docs = loader.load()bs_transformer = BeautifulSoupTransformer()docs_transformed = bs_transformer.transform_documents(docs, tags_to_extract=["span"])splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=1000, chunk_overlap=0)splits = splitter.split_documents(docs_transformed)extracted_content = extract(schema=schema, content=splits[0].page_content)pprint.pprint(extracted_content)return extracted_content
urls = ["https://ceshiren.com/"]schema = {"properties": {"title": {"type": "string"},"url": {"type": "string"},},"required": ["title", "url"],}extracted_content = scrape_with_playwright(urls, schema=schema)
总结
了解网页爬取的实现思路以及相关技术。
通过LangChain实现爬取测试人网页的标题和url。
粤ICP备14082021号
免费POC，
零成本试错
