Where RAG Fails
Lesson 3 of 4 in RAG End to End.
RAG demos beautifully and fails quietly. The failures matter more than the happy path because they all present the same way — a wrong answer — while requiring completely different fixes. The debugging discipline that separates functional teams from flailing ones is one habit: read the retrieved chunks before blaming the model. Every failure in this catalog is diagnosed by comparing three artifacts — the question, the chunks that were actually assembled into the prompt, and the answer.
Five failure modes cover most of what you will meet in production:
Retrieval miss — the right chunks never arrived. The query’s wording and the document’s wording live far apart in Embedding space, the answer is scattered across chunks none of which scores well alone, or the corpus simply lacks the answer. The generator is now doing closed-book trivia with a confident tone.
Context dilution — the right chunk arrived and drowned. Raising k feels like generosity, but every near-miss chunk is a distractor competing for the model’s attention, and models use the middle of a long context least reliably — the “lost in the middle” effect (Liu et al., 2023). Recall improves on paper while answers get worse in practice.
Grounding failure — the evidence was present, readable, and ignored. The model answers from its weights, blends memory with evidence, or contradicts the chunk outright. This is the RAG-flavored face of Hallucination, and prompting discipline (the refusal path from lesson one) is the first line of defense — the Security & Risk domain treats the model-level failure in full.
Stale index — everything works perfectly on last month’s truth. Freshness is an ingest property; a document that changed but was never re-indexed is invisible.
Chunk-boundary truncation — the chunker cut mid-thought: a rule from its exception, a table from its header, step 3 from steps 1–2. Retrieval finds the fragment, and the model grounds faithfully on half an idea — arguably the nastiest failure, because every downstream stage behaves correctly.
| Failure | Symptom | How to confirm | First fix |
|---|---|---|---|
Retrieval miss | Answer is generic, off-topic, or fabricated; citations absent or irrelevant | Read the retrieved chunks: the answer is not in them | Improve recall: hybrid search for exact terms, better chunking, query rewriting; check the corpus even contains the answer |
Context dilution | Quality dropped after raising k or adding sources; long, hedging answers | The right chunk is present but buried mid-context among near-misses | Rerank and cut: fewer, better chunks, strongest evidence first |
Grounding failure | Answer contradicts or ignores retrieved evidence; uncited claims | The right chunk is present and prominent, yet the answer diverges from it | Tighten the prompt contract: answer-only-from-evidence, mandatory citations, explicit refusal path; verify citations mechanically |
Stale index | Confidently cites outdated versions; wrong “as of” facts | Compare the cited chunk against the current source document | Re-run ingest; set re-indexing cadence from corpus change rate; index timestamps as metadata |
Chunk-boundary truncation | Answers built on half a sentence, a rule without its exception, a table without its header | The retrieved chunk visibly cuts mid-thought | Re-chunk along structural boundaries; add overlap; attach section headers to every chunk |
Interactive sorting exercise: A wrong answer arrives. You pull the retrieved chunks and compare. Classify each observation by the failure it indicates.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.