The RAG That Retrieved Nothing

A support assistant shipped with fluent, confident answers that never touched the corpus — and the three-layer diagnosis that explained why.

A composite teaching case: realistic fiction assembled from well-documented public patterns — not a real engagement.

The system was an internal support assistant: answer plan, entitlement, and policy questions for a few hundred support agents, from a corpus of product wiki pages, policy PDFs, and — the part that mattered most to agents — the entitlement tables that say exactly what each plan tier includes. Textbook Retrieval-augmented generation (RAG): an ingest pipeline parsed the corpus, cut it into fixed-size chunks, ran them through an Embedding model, and loaded the vectors into Vertex AI Vector Search; the serving path embedded each incoming question, pulled top-k candidates from the Approximate nearest neighbor (ANN) index, reranked them, and handed the survivors to a Gemini model on Vertex AI under a prompt contract: answer only from the provided context, cite chunk IDs.

The demo, in front of the support leadership, went beautifully. Launch was five weeks later. In between, launch-hardening happened: dependencies bumped, configs consolidated, secrets rotated. One of those tidy-ups moved the serving path's embedding call onto a shared config entry — which pointed at a newer embedding model than the one the ingest pipeline had used to build the index months earlier. Nobody noticed, because nothing failed. Both models returned vectors. The index accepted the queries. Every request got a 200.

The symptom reports started in week one, and they were strange because nothing about them looked broken. The assistant was fluent. Well-structured answers, confident tone, the right vocabulary, plausible caveats. Agents' complaints all had the same shape: “it sounds like our docs, but it's wrong about everything specific.” Plan names slightly off. Seat counts that belonged to no tier we sell. Policy answers that read like industry best practice instead of our policy. Citations were the tell we should have read first: answers either carried no chunk IDs at all, or cited chunks that had nothing to do with the question.

This is the signature worth memorizing: fluent, generic, confidently wrong is not what a broken generator sounds like — it is what a working generator sounds like when it has been handed no usable evidence. The model was doing closed-book trivia from its own Weights, in exactly the situation Grounding contracts exist to prevent, and dressing the result in our corpus's tone of voice. Pure Hallucination, delivered with excellent manners.

Wrong turn one: blame the model. The most visible component gets indicted first, so we swapped the Gemini model for a larger one and spent a week A/B-reading answers. The bigger model wrote noticeably better prose around exactly the same wrong facts — which we mistook for progress and shipped. Cost: a week, plus a higher per-token bill on every request while it lasted.

Wrong turn two: beg the prompt. Next we escalated the System prompt: “You MUST use ONLY the provided context.” Capital letters. Repeated instructions. Temperature to zero. And here the story turns genuinely instructive: the begging worked, in the worst possible way. The model started obeying — and since the context it was ordered to rely on was noise, it began refusing: “the provided context does not contain information about Standard plan SSO.” Refusal rates annoyed agents, so we softened the prompt again, which restored the confident fabrication. We had built a dial that traded refusals for hallucinations and mistaken it for a quality knob. No position on that dial could inject evidence that retrieval never delivered.

The debug ladder we should have climbed on day one

  1. Fluent, wrong answer

    Confident tone, generic content, missing or irrelevant citations. Resist the reflex to open the prompt file.

  2. Is the answer in the corpus at all?

    Search the source documents by hand. If the fact was never written down, no pipeline setting can retrieve it.

  3. Corpus gap — fix ingest coverage

    A documentation problem wearing an AI costume. Write the missing page; touch nothing else.

  4. Is the right chunk in the top-k candidates?

    Read the actual retrieved chunks from the serving logs. This single check would have found our bug in week one — it is where retrieval collapse becomes visible.

  5. Retrieval problem — embeddings, chunking, index

    Our bug lived here, twice over: query and corpus embedded by different models, and tables shredded mid-row by fixed-size chunking.

  6. Is the chunk intact and prominent in the assembled prompt?

    A chunk can be retrieved and then truncated, buried mid-context, or cut off from the header that gives its numbers meaning.

  7. Assembly problem — ordering, truncation, context budget

    The evidence arrived and was then mishandled between retrieval and generation.

  8. Now — and only now — suspect the generator

    Evidence present, intact, prominent — and still ignored? This is the only branch where prompt contracts, model choice, and grounding checks are the right tools. We started here. It is the last rung, not the first.

Climbing the ladder properly turned up three layered findings — and the order in which they surfaced is itself the lesson.

Layer one, visible to the naked eye: Chunking had shredded the tables. Ingest used fixed-size chunks tuned on wiki prose. Entitlement tables got cut mid-row: a chunk would hold a stripe of cells — “25 / 100 / unlimited” — with the header row (which plan? which feature?) stranded in a different chunk. Even a perfect retriever could only have delivered shards. This explained why table answers were bad. It did not explain why everything was bad.

Layer two, invisible until we diffed configs: the embedding model differed between index time and query time. The corpus had been embedded months earlier with the model pinned in the ingest pipeline. The launch-hardening config change had quietly pointed the serving path at a newer embedding model. Two models, even at the same output dimensionality, place meaning in unrelated coordinate systems — an Embedding is only comparable to vectors from the same model, same version. Cosine similarity across the two spaces is noise, so Vector Search was faithfully returning the nearest neighbors of a meaningless comparison: near-random chunks, every request, with no error anywhere. The timeline finally made sense — the flawless demo predated the config change by weeks.

Layer three, the enabling condition: we had never measured retrieval separately from generation. Our only quality signal was end-to-end — humans reading final answers. Through that lens, total retrieval collapse is indistinguishable from “the model got dumber,” which is precisely the misdiagnosis we ran with for three weeks. A Retrieval-only metric would have registered the collapse the day the config shipped. There was no such metric, so there was nothing to falsify the wrong theory.

Bar chart of retrieval recall at 5 across three configurations. As launched, with mismatched embedding models between index and query, recall is 8 percent — near random. With one embedding model pinned end to end, recall jumps to 61 percent. With the pin plus structure-aware chunking that keeps table rows with their headers, recall reaches 87 percent.

Recall@5 on our labeled set — 60 real agent questions, each mapped to the chunk(s) containing the answer — measured at three stages of the fix. Values are illustrative of this composite case, not a benchmark; the shape is the point: the embedding pin was the order-of-magnitude repair, structure-aware chunking the meaningful second step. The remaining misses were vocabulary gaps between agent phrasing and document phrasing — hybrid search territory. (illustrative — source: Recall@k methodology after Karpukhin et al. (2020) — Dense Passage Retrieval, arXiv:2004.04906; values are illustrative composite measurements)

The fix inverted our debugging order into a build order: measure retrieval first, then repair it, then — last — touch generation.

Retrieval-first evaluation. Before changing anything, we built the labeled set the figure is measured on: 60 real questions from support tickets, each mapped by an agent to the chunk(s) that contain the answer. Against it we run recall@5 (is the right evidence in the net at all?) and MRR (does it surface near the top?). These numbers need no LLM in the loop — they are fast, cheap, and deterministic, so they run in CI on every change to chunking, embedding model, or index settings. This is the two-stage evaluation discipline from RAG End to End, and it is what turns “the answers feel worse” into a number that can be wrong.

One embedding model, pinned end-to-end. The corpus was re-embedded with the model the serving path actually uses, and the model name and version now live in the index metadata like a schema version. The serving path asserts the match at startup and refuses to serve on mismatch — pinning by enforcement, not by documentation, because documentation is what we had the first time. The mechanics of why vector spaces don't mix are in RAG Mechanics.

Structure-aware chunking. Ingest now cuts along the corpus's own structure — headings, clauses, list items — with a fixed-size fallback inside very long sections. Tables are the special case that started this: each row is serialized with its header row attached, so no cell ever travels without the context that gives it meaning. Only after recall@5 stabilized did we reopen the prompt — and mostly to delete the begging. With real evidence arriving, the original, polite grounding contract worked fine, and a separate faithfulness check on generated answers (claim-by-claim support against the retrieved chunks, judged by a model) now watches the generator's half of the system on its own scorecard.

What changed permanently was less the pipeline than the reflexes around it. Retrieval metrics are now release gates: an ingest change, an embedding upgrade, or an index-settings change that drops recall@5 on the labeled set does not ship. The embedding model identifier is treated as part of the index's type signature — upgrading it is a planned migration with a re-embedding budget, not a config edit. “Read the retrieved chunks” is the mandatory first move on any RAG bug report, and the debug ladder is pinned in the runbook in pipeline order: corpus, index, candidates, assembly, generator. Swapping the chat model — our first move, that lost week — is now formally the last rung.

And the labeled set keeps growing: every escalated wrong answer becomes a new labeled query in the eval. The system that failed silently for three weeks now converts each failure into a permanent regression test — which is the cheapest thing we bought out of the whole episode, and the most valuable.