Skip to content

Latest commit

 

History

History
49 lines (38 loc) · 2.12 KB

README.md

File metadata and controls

49 lines (38 loc) · 2.12 KB

medium-analyzer

---Ingestion

  • Loading the medium blog (TextLoader)
  • Splitting the blog into smaller chunks (TextSplitter)
  • Embed the chunks and get vectors (OpenAIEmbeddings)
  • Store the embeddings in Pinecone Vectorstore (PineconeVectorstore)

---Retrival

  • Get the embeddings from Pinecone Vectorstore, Embedding User Query
  • Semantic Search (Relevant Vectors)
  • Prompt Augmentation
  • Generation

Overview

This script retrieves information using language models and vector stores based on a given query. It combines different chains to retrieve relevant documents and provide concise answers to questions.

Functions and Methods

format_docs

  • Description: Formats a list of documents into a single string, separating each document by two newline characters.
  • Parameters:
    • docs: List of documents to be formatted.
  • Returns:
    • A string containing the formatted documents.

Main Script Execution

  1. Retrieving Information:

    • The script begins by initializing language models and loading the necessary environment variables.

  2. Creating Chains:

    • Embeddings and Language Models:
      • OpenAI embeddings and ChatOpenAI language model are initialized.
    • Prompt Template:
      • A prompt template is created from the query "What is Pinecone in Machine Learning?" using the PromptTemplate class.
    • Vector Store:
      • A Pinecone vector store is initialized with the specified index name and embeddings.
    • Retrieval Chains:
      • Retrieval chains are created using the create_stuff_documents_chain and create_retrieval_chain functions, which combine language models and vector stores to retrieve relevant documents based on the query.

  3. Invoking Chains:

    • The retrieval chain is invoked with the query input, which retrieves relevant documents.
    • A custom RAG prompt template is created to provide helpful answers to the question.
    • The RAG chain is invoked with the query to generate a response using the retrieved documents and the specified question.

  4. Output:

    • The retrieved documents and the response generated by the RAG chain are printed to the console.