Troubleshooting retrieval: drift, chunks, embeddings
Lesson 2 of 4 in Troubleshooting GenAI — and Exam Readiness.
Retrieval failures wear a model costume: the assistant "hallucinates", "got dumber", "makes things up" — and the instinctive fixes (bigger model, sterner prompt) treat the actor, not the script. The discipline the AIP-C01 exam guide teaches is localize first: for a failing query, pull the actual retrieved chunks and read them. That single act of model response relevance analysis splits the universe — right chunks with a wrong answer is a generation problem (Faithfulness); wrong chunks is a retrieval problem with its own taxonomy: embedding issues, Chunking shape, index sync, ranking. The tree below is the walk.
The retrieval triage tree
Interactive decision tree — outcomes:
- The generation half is guilty
Retrieval is exonerated: the evidence arrived and the model embellished past it. Work the generation side — grounding instructions, guardrail contextual-grounding thresholds, model choice — and confirm with the faithfulness metric in a RAG evaluation re-run. Depth: engineering hallucinations down.
- An ingestion / sync gap
The corpus moved and the index did not: failed or stale sync jobs, a source added outside the pipeline, deletions never propagated. Re-ingest and alarm on sync-job failures — this is operational, not algorithmic. Depth: keeping indexes in sync.
- Embedding model mismatch
Documents embedded with one model, queries with another — two incompatible vector spaces, so relevance collapses at once. The only real fix is re-indexing the full corpus with the query-side model. Prevent it by versioning the embedding model with the index and gating embedding changes behind a RAG evaluation run.
- Embedding / corpus drift
Vocabulary moved — new products, new jargon, new question styles — until stored vectors stopped representing what users ask. Schedule re-embedding, monitor retrieval scores (context relevance over time), and treat a sustained slide as a re-index trigger. This is Embedding drift in its slow form.
- Chunks cut the answers apart
Fixed-size splitting severed semantic units, so no single chunk answers the question. Remediate with semantic or hierarchical Chunking and overlap, sized to the content type. Depth: chunking strategies.
- Chunks drown the signal
Oversized chunks match everything weakly and stuff the context window with noise (and tokens). Shrink them, then let a Reranker restore precision on the larger candidate set.
- A ranking problem
The content exists in well-shaped chunks — ordering fails it. Hybrid search (semantic + keyword), metadata filtering to cut the candidate space, a Reranker on the top-k, and query transformation for vague questions. The cheapest branch to fix.
| Symptom | Likely cause | Remediation |
|---|---|---|
Relevance collapsed overnight after an upgrade | Index and queries now use different embedding models | Re-embed and re-index the whole corpus with the query-side model; version the embedding model with the index |
Quality sliding for months as new docs arrive | Corpus/query vocabulary drift | Scheduled re-embedding; monitor context-relevance scores as a time series and alarm on sustained decline |
Multi-part questions answered half-right | Fixed-size chunking severed semantic units | Semantic or hierarchical chunking with overlap; re-run the retrieval evaluation to confirm |
Right document retrieved, wrong section ranks first | Pure vector ranking on a crowded corpus | Hybrid search + metadata filters + a Reranker over a wider top-k |
Retrieval p95 latency breaks the SLO | Index/search configuration, unfiltered scans, cold caches | Tune index parameters, filter before vector search, cache frequent queries — and keep retrieval latency on the dashboard (the guide makes it a tested metric) |
Prevention is a test harness, not vigilance. Retrieval quality testing keeps three instruments running: relevance scoring over a fixed query set (the retrieval half of RAG evaluation), context matching verification — do the retrieved chunks actually contain the expected ground-truth passages — and retrieval latency measurement, because a right answer after twelve seconds is still a failure.
Every remediation then closes with the same move you saw in the truncation hunt: re-run the identical evaluation job and compare. Chunking change, re-embedding, reranker added — the before/after pair of job scores (normalized 0–1, per metric) is the proof the fix fixed, and the pair of job IDs is the artifact you show stakeholders.
Key terms: Embedding drift, RAG evaluation, Faithfulness, Chunking, Reranker
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.