Case study

RAG Document Assistant

Jul 2026 · Personal project

I built a question-answering service for PDFs. Claude is told to answer only from retrieved passages. The server attaches the file and page behind every citation, and an answer that comes back without citations is flagged as not grounded.

  • RAG
  • FastAPI
  • ChromaDB
  • Claude API
  • Docker
  • CI
  • Evaluation
  • LLM
answerable eval questions where the right page was retrieved
13/15
unanswerable eval questions counted as abstentions
5/5
mean time per answer in the eval run
4.1 s
offline tests, run in CI
29

The first three numbers come from one eval run: 20 questions about a single 70-page report. Read them as a sanity check, not a benchmark.

What an answer looks like

Two exchanges from the demo recording in the README, asked against a 70-page engineering report. The first answer cites its pages. The second question is not covered by the report, so the assistant says so instead of guessing.

Question: What camera module was selected for the microscope, and what is its resolution in megapixels?

Answer: The Raspberry Pi HQ Camera was selected for the microscope. It has a high resolution of 12.3 megapixels.

  • dhm_report.pdf, p. 38
  • dhm_report.pdf, p. 63

Question: What's the capital of Mongolia?

Not grounded in the documents

Answer: The provided sources do not contain any information about the capital of Mongolia.

How it works

  1. Ingest

    pypdf extracts the text page by page. Each page is split into overlapping chunks of up to 1,200 characters that keep their file name and page number.

  2. Index

    Chunks are embedded locally with all-MiniLM-L6-v2 and stored in a persistent ChromaDB collection that searches by cosine distance.

  3. Retrieve

    For each question the closest chunks (10 by default) are retrieved and numbered as sources.

  4. Generate

    Claude is instructed to use only those sources and to back every claim with a citation and a short quote. The reply must match a JSON schema.

  5. Resolve

    The server maps each citation number to the file and page of the retrieved chunk. An answer with no citations is flagged as not grounded.

  6. Serve

    A FastAPI service handles upload, listing, deletion and questions. A Streamlit chat UI calls it and shows the citations under each answer.

The problem

Long documents such as clinical guidelines, contracts and papers are risky to summarize from memory. A language model answering from its own knowledge can invent a dosage or a clause.

I wanted a service that answers from the uploaded documents and shows where each answer came from, so a reader can check it. When the documents do not hold the answer, it should say so.

Keeping answers grounded

Three layers keep answers tied to the documents.

  • Prompt contract. Claude may use only the numbered sources, must cite one for every claim, and must say so when the sources are not enough.
  • Structured output. The reply has to match a JSON schema with an answer and a list of citations, so the server never parses free text.
  • Server-side resolution. Citation numbers are mapped to real files and pages. If nothing is retrieved, the service returns a not-grounded message without calling the model.

Results

I tested it on 20 questions about one 70-page engineering report, which I co-wrote for my undergraduate Digital Holographic Microscope project. The report answers 15 of them and does not answer the other 5.

For 13 of the 15 answerable questions the expected page was among the retrieved chunks. All 5 unanswerable questions were counted as abstentions: the answer came back with no citations or said the sources did not state the fact. Mean time per answer was 4.1 seconds.

The two misses taught me the most. In one, the chunk with the answer ranked 11th because an unrelated paragraph at its start dominated its embedding. In the other, the model answered confidently, with valid citations, from a background passage about an alternative setup that the project did not use. Grounded means the answer came from the document. It does not mean it came from the right passage.

Testing and delivery

29 tests cover chunking, the vector store, the LLM contract and the API. They swap in a deterministic embedding and a fake Claude client, so they run offline with no API key and no model download.

GitHub Actions runs Ruff and pytest on every push to main and on pull requests. Docker Compose starts the API and the UI with one command and keeps the data in a named volume.

Limits and next steps

  • Scanned PDFs are rejected. There is no OCR step.
  • Chunking is by characters, not by tokens or layout, so tables and headers get no special handling.
  • The quote inside a citation is written by the model and is not yet string-checked against the chunk.
  • One shared index with no sign-in or per-user isolation. It is a demo, not a multi-tenant service.
  • The eval is small and hand-built: 20 questions about one document.
  • Next: quote verification, hybrid BM25 plus dense retrieval with reranking, OCR fallback, streaming answers and LLM-as-judge scoring.

Built with

Model
  • Claude (Anthropic SDK)
  • JSON-schema output
  • all-MiniLM-L6-v2 embeddings
Retrieval and data
  • ChromaDB
  • pypdf
API and UI
  • FastAPI
  • Pydantic
  • Uvicorn
  • Streamlit
Tooling
  • pytest
  • Ruff
  • GitHub Actions
  • Docker Compose

Let's build something people love.

I'm open to ML/AI engineering and data science roles, collaborations, and interesting projects. The fastest way to reach me is email.