RAG with ollama (and optionally cohere) and qdrant. This is basically a glorified
(bloated) grep
.
Create a .env file or set the following parameters:
CHUNK_SIZE=4096
CHUNK_OVERLAP=256
ENCODER_MODEL=nomic-embed-text
EMBEDDING_DIM=768
RETRIEVER_TOP_K=15
RETRIEVER_SCORE_THRESHOLD=0.5
RERANK_MODEL=mixedbread-ai/mxbai-rerank-large-v1
RERANK_TOP_K=5
GENERATOR_MODEL=llama3
DOCUMENT_DB_NAME=rag
DOCUMENT_DB_USER=aktersnurra
QDRANT_URL=http://localhost:6333
QDRANT_COLLECTION_NAME=knowledge-base
COHERE_API_KEY = <COHERE_API_KEY> # OPTIONAL
COHERE_RERANK_MODEL = "rerank-english-v3.0"
poetry install
Make sure ollama is running:
ollama serve
Download the encoder and generator models with ollama:
ollama pull $GENERATOR_MODEL
ollama pull $ENCODER_MODEL
Qdrant is used to store the embeddings of the chunks from the documents.
Download and run qdrant.
Postgres is used to save hashes of the document to prevent documents from being added to the vector db more than ones.
Download and run qdrant.
Get an API from their website, but is optional.
Activate the poetry shell:
poetry shell
Use the cli:
python rag/cli.py
or the ui using a browser:
streamlit run rag/ui.py
Yes, it is inefficient/dumb to use ollama when you can just load the models with python in the same process.
-
Rerank history if it is relevant.
-
message ollama/cohere
-
create db script
-
write a general model for cli/ui
-
use huggingface instead of ollama
-
Refactor messages
I took some inspiration from these tutorials:
building-rag-application-using-langchain-openai-faiss knowledge_gpt