Evaluating RAG
Lesson 4 of 4 in RAG End to End.
Everything in the failure-modes lesson pointed at one conclusion: RAG is a two-component system whose components fail independently, so it must be evaluated as two components. A single end-to-end “answer quality” number cannot tell you whether to fix chunking or fix the prompt — and teams with only that number fix the wrong one for weeks.
Stage one: evaluate retrieval alone. Build a labeled set: real queries, each mapped to the chunk(s) or document(s) that actually contain the answer. Then measure, at the concept level: recall@k — for what fraction of queries does the correct evidence appear anywhere in the top k results? — and MRR (mean reciprocal rank) — how high does the first correct result sit, on average? Recall@k tells you whether the net catches the fish; MRR tells you whether it surfaces near the top, which matters because assembled context is ordered and, as the reranking lesson showed, position decides salience. These metrics run without any Large language model (LLM) in the loop: fast, cheap, deterministic — you can run them on every change to chunking, embedding model, or index settings.
Stage two: evaluate generation against the evidence. Given the retrieved chunks and the generated answer, ask: is every claim in the answer supported by the evidence — is the answer grounded (faithful)? Note what this is not: it is not “is the answer correct?” An answer can be perfectly faithful to retrieved evidence that was wrong or stale, and correct by luck while citing nothing. Faithfulness isolates the generator’s contribution — exactly the failure that prompt contracts are supposed to prevent. Claim-by-claim support checks are commonly automated with a judge model; the methodology for doing that defensibly — judges, rubrics, their biases — is the Evaluation domain’s territory.
Two-stage RAG evaluation — retrieval measured without the generator, generation measured against the evidence
- Labeled eval set
Real queries mapped to the evidence that answers them. Built from production logs and support tickets — not from questions invented while staring at the documents.
- Run retrieval only
For each query, record the top-k retrieved chunks. No generation, no LLM — cheap enough to run on every pipeline change.
- Score: recall@k, MRR
Did the correct evidence appear in the top k (recall@k)? How high did it rank (MRR)? These localize failures to the ingest/retrieval side.
- Retrieval healthy?
If recall is low, stop — fix chunking, embeddings, or search first. Generation metrics are meaningless over evidence that is not there.
- Generate answers over retrieved evidence
Run the full serving path so the generator sees exactly what production would assemble.
- Score: groundedness / faithfulness
Is every claim in the answer supported by the retrieved chunks? Checked claim by claim — commonly automated with a judge model (Evaluation domain).
- Two scores, two dials
Retrieval score moves → fix ingest/search. Groundedness moves → fix prompt contract or model choice. One end-to-end number cannot make this distinction.
| Metric | Stage it isolates | Question it answers | When it drops, look at |
|---|---|---|---|
Recall@k | Retrieval | Is the correct evidence anywhere in the top k? | Chunking, embedding model, hybrid search, corpus coverage, index freshness |
MRR | Retrieval + reranking | How high does the first correct chunk rank? | Reranker quality, candidate depth, near-duplicate chunks crowding the top |
Groundedness / faithfulness | Generation | Is every claim in the answer supported by the retrieved evidence? | Prompt contract, citation enforcement, model choice, context dilution |
End-to-end answer quality | Everything at once — confounded | Did the user get a good answer? | Useful as a headline; useless for diagnosis until split into the rows above |
One more thing decides whether any of these numbers mean anything: where the eval set comes from. Build it from real queries — production logs, support tickets, the questions users actually asked and the documents that actually answered them. Questions invented while staring at the corpus inherit the corpus’s vocabulary, so they retrieve suspiciously well and overstate every metric; real users paraphrase, abbreviate, use last year’s product names, and ask things the corpus half-covers. Fifty honestly labeled real queries beat five hundred synthetic ones. Keep the set versioned, grow it from every production failure you diagnose (each failure-modes incident is a free eval case), and re-run it on every pipeline change — chunking tweaks included, because they silently move retrieval metrics. The deeper harness machinery — judge calibration, statistical significance, contamination — is the Evaluation domain’s subject; the two-stage split above is the part that is specifically RAG.
In production
All three clouds ship managed RAG: point the service at your documents and it runs the whole pipeline from this module — parsing, chunking, embedding, indexing, retrieval, and grounded generation with citations. The trade is the same everywhere: you hand over the pipeline’s knobs in exchange for not operating it. (The site’s cloud fact-table tool tracks the specific offerings.)
AWS
Amazon Bedrock’s knowledge-base capability implements ingest-to-grounded-answer as a managed flow: you supply the document source and it manages chunking, embedding, vector storage, and retrieval, returning answers with source attributions. Chunking strategy and embedding model are configurable within the menu the service offers — when your failure diagnosis calls for a custom parser or an exotic reranker, you are back to composing the pipeline yourself from the primitives.
Azure
Azure ships the same shape as “on your data” experiences and integrated vectorization in Azure AI Search: managed ingest into a search index combining vector, lexical, and reranking stages, then grounded generation over the results. The two-stage funnel from this module is visible in the service’s architecture — a recall-oriented search stage feeding a precision-oriented semantic reranking stage — with the candidate depths and models chosen by the platform.
Google Cloud
Google Cloud’s Vertex AI offers managed RAG engines and grounding services that connect models to your indexed corpora (or to web search) and return citation-annotated answers. The evaluation lesson still applies unchanged: managed or not, you own the labeled eval set and the two-stage measurement, because the provider cannot know which of your queries failed or why.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.