Case study
RAG Document Assistant
I built a question-answering service for PDFs. Claude is told to answer only from retrieved passages. The server attaches the file and page behind every citation, and an answer that comes back without citations is flagged as not grounded.
- RAG
- FastAPI
- ChromaDB
- Claude API
- Docker
- CI
- Evaluation
- LLM
- answerable eval questions where the right page was retrieved
- 13/15
- unanswerable eval questions counted as abstentions
- 5/5
- mean time per answer in the eval run
- 4.1 s
- offline tests, run in CI
- 29
The first three numbers come from one eval run: 20 questions about a single 70-page report. Read them as a sanity check, not a benchmark.
What an answer looks like
Two exchanges from the demo recording in the README, asked against a 70-page engineering report. The first answer cites its pages. The second question is not covered by the report, so the assistant says so instead of guessing.
Question: What camera module was selected for the microscope, and what is its resolution in megapixels?
Answer: The Raspberry Pi HQ Camera was selected for the microscope. It has a high resolution of 12.3 megapixels.
- dhm_report.pdf, p. 38
- dhm_report.pdf, p. 63
Question: What's the capital of Mongolia?
Not grounded in the documents
Answer: The provided sources do not contain any information about the capital of Mongolia.
How it works
Ingest
pypdf extracts the text page by page. Each page is split into overlapping chunks of up to 1,200 characters that keep their file name and page number.
Index
Chunks are embedded locally with all-MiniLM-L6-v2 and stored in a persistent ChromaDB collection that searches by cosine distance.
Retrieve
For each question the closest chunks (10 by default) are retrieved and numbered as sources.
Generate
Claude is instructed to use only those sources and to back every claim with a citation and a short quote. The reply must match a JSON schema.
Resolve
The server maps each citation number to the file and page of the retrieved chunk. An answer with no citations is flagged as not grounded.
Serve
A FastAPI service handles upload, listing, deletion and questions. A Streamlit chat UI calls it and shows the citations under each answer.
The problem
Long documents such as clinical guidelines, contracts and papers are risky to summarize from memory. A language model answering from its own knowledge can invent a dosage or a clause.
I wanted a service that answers from the uploaded documents and shows where each answer came from, so a reader can check it. When the documents do not hold the answer, it should say so.
Keeping answers grounded
Three layers keep answers tied to the documents.
- Prompt contract. Claude may use only the numbered sources, must cite one for every claim, and must say so when the sources are not enough.
- Structured output. The reply has to match a JSON schema with an answer and a list of citations, so the server never parses free text.
- Server-side resolution. Citation numbers are mapped to real files and pages. If nothing is retrieved, the service returns a not-grounded message without calling the model.
Results
I tested it on 20 questions about one 70-page engineering report, which I co-wrote for my undergraduate Digital Holographic Microscope project. The report answers 15 of them and does not answer the other 5.
For 13 of the 15 answerable questions the expected page was among the retrieved chunks. All 5 unanswerable questions were counted as abstentions: the answer came back with no citations or said the sources did not state the fact. Mean time per answer was 4.1 seconds.
The two misses taught me the most. In one, the chunk with the answer ranked 11th because an unrelated paragraph at its start dominated its embedding. In the other, the model answered confidently, with valid citations, from a background passage about an alternative setup that the project did not use. Grounded means the answer came from the document. It does not mean it came from the right passage.
Testing and delivery
29 tests cover chunking, the vector store, the LLM contract and the API. They swap in a deterministic embedding and a fake Claude client, so they run offline with no API key and no model download.
GitHub Actions runs Ruff and pytest on every push to main and on pull requests. Docker Compose starts the API and the UI with one command and keeps the data in a named volume.
Limits and next steps
- Scanned PDFs are rejected. There is no OCR step.
- Chunking is by characters, not by tokens or layout, so tables and headers get no special handling.
- The quote inside a citation is written by the model and is not yet string-checked against the chunk.
- One shared index with no sign-in or per-user isolation. It is a demo, not a multi-tenant service.
- The eval is small and hand-built: 20 questions about one document.
- Next: quote verification, hybrid BM25 plus dense retrieval with reranking, OCR fallback, streaming answers and LLM-as-judge scoring.
Built with
- Model
- Claude (Anthropic SDK)
- JSON-schema output
- all-MiniLM-L6-v2 embeddings
- Retrieval and data
- ChromaDB
- pypdf
- API and UI
- FastAPI
- Pydantic
- Uvicorn
- Streamlit
- Tooling
- pytest
- Ruff
- GitHub Actions
- Docker Compose