Retrieval in one lesson
Lesson 1 of 5 in RAG for Agents: From One-Shot Retrieval to Agentic Search.
An LLM knows nothing about your company’s refund policy, last week’s incident reports, or the contract you signed yesterday. Its knowledge froze at training time, and your private corpus was never in it. You could paste documents into the prompt — but the context window is finite, you pay for every token, and most of the corpus is irrelevant to any single question.
Retrieval-augmented generation is the standard answer: store the corpus in a searchable index, and at question time fetch only the passages that matter, put them in the context, and have the model answer from them. For an agent, retrieval is just another tool — a search call in the agent loop — which is why this module sits in the architectures track and not in a database course.
Everything downstream depends on two decisions made before any question arrives.
Chunking splits documents into retrievable pieces. Too small, and a chunk loses the context that made it meaningful — a table row without its header, step 4 of a procedure without steps 1–3. Too large, and one relevant sentence drags in two pages of noise that dilutes the context and the relevance signal. There is no universal right size; there is only tested on your corpus.
Embeddings turn each chunk into a point in a high-dimensional space where texts with similar meaning land near each other — no keyword overlap required. ‘How do I get my money back?’ lands near ‘Reimbursement procedure’ even though they share no words. A vector index stores those points and, given an embedded query, returns the nearest chunks — the top-k results.
| Mode | How it matches | Shines when | Fails when |
|---|---|---|---|
Keyword (lexical, e.g. BM25) | Exact and near-exact word overlap, weighted by how rare the words are | Queries contain identifiers the corpus also contains: error codes, part numbers, names, | The user’s words differ from the document’s words — ‘money back’ never matches ‘reimbursement’ |
Vector (semantic) | Nearness in embedding space — similarity of meaning, learned by the embedding model | Paraphrase and synonym gaps: the question is phrased nothing like the answer text | Exact tokens matter: rare IDs, version numbers, and codes embed poorly and get lost among ‘similar-ish’ chunks |
Hybrid (both, scores merged) | Runs keyword and vector in parallel and fuses the ranked lists | Real production traffic — which is always a mix of identifier-style and natural-language queries | Rarely worse than either alone; costs two searches and a fusion step, and still inherits chunking mistakes |
… + reranking (second stage) | A slower, smarter model reads each (query, candidate) pair from the first stage’s top ~50 and re-orders them | Precision at the top matters — only 3–5 chunks will fit the prompt, so the order is everything | Latency budgets are tight: it adds a model pass per candidate, and it cannot rescue documents the first stage never fetched |
Read that last row again, because it states the pattern of this whole module: first-stage retrieval is cheap and coarse; reranking is expensive and sharp; and no later stage can recover what an earlier stage dropped. Fetch wide, then rank hard, then hand the model only what earned its place in the context.
Key terms: RAG, chunking, embedding, vector search, hybrid search, reranking
Interactive sorting exercise: Which retrieval mode is the best *first choice* for each query? (Assume you can only pick one.)
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.