Recently, with OpenAI releasing its API, there has been a surge in AI applications in the market. VeryPDF now offers customized development services for ChatPDF. ChatPDF is a tool that allows you to interact with PDF files, enabling quick extraction of information from documents such as manuals, papers, contracts, books, and more.
Due to limitations on token length in the OpenAI API, VeryPDF has utilized special techniques to successfully surpass the API's maximum token limit for reading lengthy texts.
Overview of ChatPDF:
- Extract text from PDF for further processing.
- As the OpenAI API has token limitations, we need to segment the PDF text into fragments smaller than the token limit.
- Generate vectors for each segment using OpenAI's Embedding API and store them in a database (Postgres, MySQL or others).
- Begin asking questions.
- Convert user-posed questions into vectors.
- Employ the cosine similarity algorithm to compare the user's question vector with vectors in the database to find the most similar text fragment.
- Feed the text fragment to ChatGPT to generate responses based on these fragments.
This explanation focuses on the general principles. When delving into the code, specific aspects include:
- How to extract text?
- What criteria are used for segmentation? How to ensure each segment is semantically related as much as possible?
Since the extracted PDF text comprises words, segmentation is done based on the number of words and "sentences" are used as dimensions for segmentation, separating text at periods and line breaks. If it's a Markdown format, segmentation becomes simpler, done based on paragraphs, ensuring semantic coherence in each segment (the task of semantic separation is essentially delegated to the Markdown writer).
Technological Stack Used:
- PostgresSql
- Next.js
- Supabase: Used for storing vectors and text fragments.
Presently, due to restricted usage of the OpenAI API, controlling the interface call frequency is essential for generating vectors from large PDF files.
-- Proper Nouns
Here are some proper nouns. I'll provide a brief explanation according to ChatGPT's understanding of these terms:
-- Embedding
Embedding is a technique to transform discrete data (such as words, characters, images, etc.) into continuous vectors. In natural language processing, Embedding enables mapping of words or characters into a lower-dimensional continuous vector space, effectively representing semantic information. For instance, words like "cat" and "dog" might be mapped close to each other in the Embedding space since they represent animals, while "cat" and "table" might be mapped far apart since they represent different things.
In the ChatGPT PDF project, we use OpenAI's Embedding API to convert PDF text fragments into vectors and store these vectors in a database. This approach helps better represent the semantic information of text fragments, enhancing the accuracy of question matching.
-- Cosine Similarity Algorithm
The cosine similarity algorithm calculates the similarity between two vectors by measuring the cosine of the angle between them.
In the ChatGPT PDF project, we initially compute the cosine similarity between the user's question vector and each text fragment vector in the database. Then, we select the most similar text fragment as the context for posing questions to ChatGPT.
If you're interested in the ChatPDF project and wish to engage VeryPDF for the development of this software, please get in touch with us. We'll assist you in bringing this project to life.