Personal project
Chat with your documents (RAG)
A retrieval-augmented generation system that answers questions from a document collection with local models, grounding answers in the documents and removing LLM API costs.
- Role
- Designed and built
- When
- 2025
- answer accuracy vs. a standalone LLM
- +45%
- hallucinations, through retrieved context
- −60%
- local setup time with Docker
- 45 → 2 min
What it does
Answers questions about a collection of documents by retrieving relevant passages and giving them to a local LLM as context, so answers come from the documents rather than the model’s memory.
What I built
- A RAG pipeline with Ollama embeddings and a Chroma vector store, built with LangChain.
- Local inference with Gemma 3 12B, which removed LLM API costs entirely.
- Custom prompt engineering and context injection, which improved response relevance by 35%.
- A Docker setup that cut local setup from 45 minutes to 2 minutes, replaced 15+ manual install steps, and gave consistent environments across 3 development machines, resolving 5+ “works on my machine” issues.
Results
- 45% higher answer accuracy than a standalone LLM.
- 60% fewer hallucinations, from grounding answers in retrieved documents.