Evaluate retrieval before you blame the model

Lesson 5 of 5 in RAG for Agents: From One-Shot Retrieval to Agentic Search.

A RAG system is two systems wearing one trench coat: a retriever and a generator. They fail independently — and if the retriever never fetched the passage containing the answer, no prompt, no model upgrade, and no amount of ‘be more accurate’ will save the generation step. Yet teams routinely respond to a wrong answer by tweaking the prompt, because the prompt is the part they can see.

The discipline is to evaluate the stages separately. Retrieval alone: for a set of questions with known source passages (a golden dataset), did the right passage appear in the top-k? Generation alone: given the right passages, is the answer correct and grounded? Two numbers, two very different fixes.

Debugging a wrong RAG answer

  1. Answer is wrong
  2. Was the answer-bearing passage in the retrieved set?

    Look at the trace: the k chunks actually handed to the model. This single check splits the problem in half.

  3. Retrieval problem

    Fix chunking, switch to hybrid, tune top-k, add reranking, or let an agent reformulate the query. The generator was never in a position to succeed.

  4. Generation problem

    The evidence was there and the model ignored, blended, or over-claimed past it. Now grounding checks, prompting, and model choice are the right levers.

  5. Fix the index / search, re-measure retrieval
  6. Fix grounding / prompt, re-measure generation

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.

Interactive flashcard deck.

Tool: Eval Suite Builder — Ready to measure a retrieval pipeline for real? The eval builder walks you through golden datasets, retrieval metrics, and groundedness judges.