使用大型语言模型提取pdf文档元数据信息

project | |---Extract_Metadata_With_Large_Language_Models.ipynb |data | |---- extracted_metadata/ |---- 1706.03762v7.pdf |---- 1810.04805.pdf |---- prompts | |------ scientific_papers_prompt.txt

Scientific research paper:---{document}---
You are an expert in analyzing scientific research papers. Please carefully read the provided research paper above and extract the following key information:
Extract these six (6) properties from the research paper:- Paper Title: The full title of the research paper- Publication Year: The year the paper was published- Authors: The full names of all authors of the paper- Author Contact: A list of dictionaries, where each dictionary contains the following keys for each author:- Name: The full name of the author- Institution: The institutional affiliation of the author- Email: The email address of the author (if provided)- Abstract: The full text of the paper's abstract- Summary Abstract: A concise summary of the abstract in 2-3 sentences, highlighting the key points
Guidelines:- The extracted information should be factual and accurate to the document.- Be extremely concise, except for the Abstract which should be copied in full.- The extracted entities should be self-contained and easily understood without the rest of the paper.- If any property is missing from the paper, please leave the field empty rather than guessing.- For the Summary Abstract, focus on the main objectives, methods, and key findings of the research.- For Author Contact, create an entry for each author, even if some information is missing. If an email or institution is not provided for an author, leave that field empty in the dictionary.
Answer in JSON format. The JSON should contain 6 keys: "PaperTitle", "PublicationYear", "Authors", "AuthorContact", "Abstract", and "SummaryAbstract". The "AuthorContact" should be a list of dictionaries as described above.

Scientific research paper:---{document}---

You are an expert in analyzing scientific research papers.

Extract these six (6) properties from the research paper:

Answer in JSON format. The JSON should contain 6 keys: ...

%%bash
pip -qqq install pdfminer.sixpip -qqq install pillow-heif==0.3.2pip -qqq install matplotlibpip -qqq install unstructured-inferencepip -qqq install unstructured-pytesseractpip -qqq install tesseract-ocrpip -qqq install unstructuredpip -qqq install openaipip -qqq install PyPDF2
apt install -V tesseract-ocrapt install -V libtesseract-dev
sudo apt-get updateapt-get install -V poppler-utils

import osimport reimport jsonimport openaifrom pathlib import Pathfrom openai import OpenAIfrom PyPDF2 import PdfReaderfrom google.colab import userdatafrom unstructured.partition.pdf import partition_pdffrom tenacity import retry, wait_random_exponential, stop_after_attempt

OPENAI_API_KEY = userdata.get('OPEN_AI_KEY')model_ID = userdata.get('GPT_MODEL')os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY
client = OpenAI(api_key = OPENAI_API_KEY)

def extract_text_from_pdf(pdf_path: str):"""Extract text content from a PDF file using the unstructured library."""elements = partition_pdf(pdf_path, strategy="hi_res")return "\n".join([str(element) for element in elements])

def read_prompt(prompt_path: str):"""Read the prompt for research paper parsing from a text file."""with open(prompt_path, "r") as f:return f.read()

@retry(wait=wait_random_exponential(min=1, max=120), stop=stop_after_attempt(10))def completion_with_backoff(**kwargs):return client.chat.completions.create(**kwargs)

def extract_metadata(content: str, prompt_path: str, model_id: str):"""Use GPT model to extract metadata from the research paper content based on the given prompt."""prompt_data = read_prompt(prompt_path)
try:response = completion_with_backoff(model=model_id,messages=[{"role": "system", "content": prompt_data},{"role": "user", "content": content}],temperature=0.2,)
response_content = response.choices[0].message.content# Process and return the extracted metadata# ...except Exception as e:print(f"Error calling OpenAI API: {e}")return {}

def process_research_paper(pdf_path: str, prompt: str, output_folder: str, model_id: str):"""Process a single research paper through the entire pipeline."""print(f"Processing research paper: {pdf_path}")
try:# Step 1: Extract text content from the PDFcontent = extract_text_from_pdf(pdf_path)
# Step 2: Extract metadata using GPT modelmetadata = extract_metadata(content, prompt, model_id)
# Step 3: Save the result as a JSON fileoutput_filename = Path(pdf_path).stem + '.json'output_path = os.path.join(output_folder, output_filename)
with open(output_path, 'w') as f:json.dump(metadata, f, indent=2)print(f"Saved metadata to {output_path}")
except Exception as e:print(f"Error processing {pdf_path}: {e}")

# Example for a single document
pdf_path = "./data/1706.03762v7.pdf"prompt_path ="./data/prompts/scientific_papers_prompt.txt"output_folder = "./data/extracted_metadata"
process_research_paper(pdf_path, prompt_path, output_folder, model_ID)

{"PaperTitle": "Attention Is All You Need","PublicationYear": "2017","Authors": ["Ashish Vaswani","Noam Shazeer","Niki Parmar","Jakob Uszkoreit","Llion Jones","Aidan N. Gomez","Lukasz Kaiser","Illia Polosukhin"],"AuthorContact": [{"Name": "Ashish Vaswani","Institution": "Google Brain","Email": "avaswani@google.com"},{"Name": "Noam Shazeer","Institution": "Google Brain","Email": "noam@google.com"},{"Name": "Niki Parmar","Institution": "Google Research","Email": "nikip@google.com"},{"Name": "Jakob Uszkoreit","Institution": "Google Research","Email": "usz@google.com"},{"Name": "Llion Jones","Institution": "Google Research","Email": "llion@google.com"},{"Name": "Aidan N. Gomez","Institution": "University of Toronto","Email": "aidan@cs.toronto.edu"},{"Name": "Lukasz Kaiser","Institution": "Google Brain","Email": "lukaszkaiser@google.com"},{"Name": "Illia Polosukhin","Institution": "","Email": "illia.polosukhin@gmail.com"}],"Abstract": "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.","SummaryAbstract": "The paper introduces the Transformer, a novel network architecture based solely on attention mechanisms, eliminating the need for recurrence and convolutions. The Transformer achieves superior performance on machine translation tasks, setting new state-of-the-art BLEU scores while being more parallelizable and requiring less training time. Additionally, it generalizes well to other tasks such as English constituency parsing."}

The paper introduces the Transformer, a novel network architecture based solely on attention mechanisms, eliminating the need for recurrence and convolutions. The Transformer achieves superior performance on machine translation tasks, setting new state-of-the-art BLEU scores while being more parallelizable and requiring less training time. Additionally, it generalizes well to other tasks such as English constituency parsin

# Parse documents from a folderdef process_directory(prompt_path: str, directory_path: str, output_folder: str, model_id: str):"""Process all PDF files in the given directory."""
# Iterate through all files in the directoryfor filename in os.listdir(directory_path):if filename.lower().endswith('.pdf'):pdf_path = os.path.join(directory_path, filename)process_research_paper(pdf_path, prompt_path, output_folder, model_id)

# Define pathsprompt_path = "./data/prompts/scientific_papers_prompt.txt"directory_path = "./data"output_folder = "./data/extracted_metadata"

process_directory(prompt_path, directory_path, output_folder, model_ID)