The Retrieval Idea

Lesson 1 of 4 in RAG Mechanics: Embeddings and Search.

The context-engineering module ended on a cliffhanger: the model only knows what is in its weights — frozen at the Knowledge cutoff — or in its Context window, and the window is a budget. Grounding an answer in your documents means getting the right few thousand tokens of evidence into that budget, per question, out of a corpus that may hold millions of pages. Deciding what deserves the window is a search problem, and this module is about the machinery that solves it.

The pattern has a name: retrieval-augmented generation — retrieve evidence, then generate conditioned on it (Lewis et al., 2020, gave the pattern its name). The sister AI Agent Academy teaches when and how an agent should invoke retrieval; here we go under the hood of the search itself, because when RAG disappoints, the bug is almost always down here.

The core move of modern Retrieval is to turn search into geometry. An Embedding model maps any piece of text to a vector — a list of a few hundred to a few thousand numbers — trained so that texts with related meanings land near each other. Do that once, offline, for every chunk of your corpus, and store the vectors in an index. At query time, embed the question with the same model and ask the index a purely mathematical question: which stored vectors are nearest? Those nearest neighbors are your candidate evidence. No keywords required — the query “why can’t staff log in?” can land next to a passage about authentication outages that shares almost no words with it.

Dense retrieval: the offline and online halves

  1. Documents (ingest time)

    Wikis, PDFs, tickets, code — whatever corpus should ground the model’s answers.

  2. Chunk and embed

    Split documents into retrieval-sized chunks (lesson 3), then run each chunk through the embedding model once.

  3. Vector index

    Chunk vectors plus metadata, stored in a structure built for fast nearest-neighbor search (lesson 4).

  4. User question (query time)

    Arrives at request time — the only part of the loop you cannot precompute.

  5. Embed the query

    One forward pass through the same embedding model, same version, that embedded the corpus.

  6. Nearest-neighbor search

    Compare the query vector against the index — exactly for small corpora, approximately (ANN) at scale.

  7. Top-k candidate chunks

    A shortlist of likely evidence, ranked by similarity — candidates, not yet answers.

  8. Into the model’s context

    The candidates (often after reranking — next module) become the evidence section of the prompt.

Dense vectors are not the only signal, and they were not the first. The classic lexical baseline is BM25: score each document by how many query terms it contains, weighted so that rare terms count for more and very long documents do not win by sheer bulk. It runs on an inverted index — the data structure behind every traditional search engine — and it is still formidable. Dense retrieval earned its place by beating strong lexical baselines on open-domain question answering with learned passage embeddings (dense passage retrieval — Karpukhin et al., 2020), precisely on the queries lexical search fumbles: paraphrase, synonyms, questions that share no vocabulary with their answers.

But each signal fails where the other is strong. Dense embeddings compress a whole chunk into one vector, so a rare identifier — an error code, a part number, a person’s name — barely dents the geometry, while BM25 treats that same rare term as gold. Production systems therefore usually run both and fuse the rankings: Hybrid search, which lesson 4 takes apart. Keep one more distinction sharp: retrieval returns candidates, not answers. The shortlist gets reranked, filtered, and finally read by the Large language model (LLM) — and a wrong shortlist cannot be fixed by any amount of clever prompting downstream.

Where the candidates land: a grounded-answer scaffold
Answer the question using ONLY the evidence below.
If the evidence is insufficient, say so explicitly — do not guess.

Evidence:
{{RETRIEVED_CHUNKS}}

Each chunk is labeled [source: {{DOC_ID}} §{{SECTION}}]. Cite the label
for every claim you make.

Question: {{QUESTION}}

The destination of everything this module builds: retrieved chunks become the evidence section of a prompt like this. The instruction to admit insufficiency is your seatbelt against confident hallucination when retrieval misses — the RAG End to End module dissects that failure mode.

Key terms: Retrieval-augmented generation (RAG), Retrieval, Embedding model, Chunking, Approximate nearest neighbor (ANN), Hybrid search

Tool: Embedding Space Explorer — Feel the geometry yourself: type queries and passages into the Embedding Explorer and watch which neighbors surface — including the near-misses that make retrieval interesting.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.