Reranking
Lesson 2 of 4 in RAG End to End.
Vector search has a dirty secret: it is optimized for finding relevant chunks, not for ordering them. A bi-encoder — the architecture behind every Embedding model used for retrieval, in the lineage of Sentence-BERT (Reimers & Gurevych, 2019) and dense passage retrieval (Karpukhin et al., 2020) — embeds the query and each document separately, then compares the frozen vectors. That separation is what makes search over millions of chunks affordable: document vectors are computed once, offline, and Approximate nearest neighbor (ANN) search does the comparison in milliseconds. But it also means the model scoring relevance never actually reads the query and the document together. Each side is compressed into a single vector before the other side is known, so fine distinctions — the right product version, negation, “does not apply when…” — blur.
A Cross-encoder fixes exactly this. It takes the query and one candidate chunk as a single concatenated input and runs a full forward pass over the pair, with Attention flowing between query tokens and document tokens. It outputs one number: how relevant is this document to this query. That joint reading is dramatically better at final ordering — and completely unusable for search, because nothing can be precomputed. Every query would need one forward pass per document in the corpus.
So production retrieval is a two-stage funnel: a cheap, precomputable stage optimized for recall casts a wide net, and an expensive, precise stage reorders the catch. You pay the cross-encoder’s cost only where it can change the outcome — the top few dozen candidates.
Two-stage retrieval: recall first, precision second (counts are generic illustrations)
- Corpus — millions of chunks
All vectors precomputed offline by the bi-encoder. The counts in this diagram are generic teaching numbers — tune yours empirically.
- Stage 1 — bi-encoder + ANN search
Embed the query once, find nearest neighbors. Cheap per query, tuned for recall: the goal is that the right chunk is somewhere in the net.
- ~50 candidates
A wide net, deliberately larger than what the context needs. Ordering within these 50 is only approximate.
- Stage 2 — cross-encoder rerank
One forward pass per query–candidate pair — 50 passes, not 50 million. Reads each pair jointly and produces a true relevance ordering.
- Top 3–5 chunks
Only the strongest survivors enter the prompt — precision protects the context from dilution.
- Into the assembled context
Fewer, better, well-ordered chunks beat many mediocre ones — the failure-modes lesson shows why.
The cost shape is worth internalizing, because it decides the whole architecture. Bi-encoder retrieval cost is dominated by offline work (embed the corpus once per document version) plus a tiny per-query cost (embed one query, walk one index). Cross-encoder cost is linear in candidates per query and zero offline — there is nothing to precompute. Rerank 50 candidates and you buy roughly 50 small forward passes per query; rerank the corpus and you buy millions. That is why the funnel narrows where it does: stage 1 exists so stage 2 stays affordable, and stage 2 exists so stage 1’s sloppy ordering never reaches the prompt.
The candidate count is your knob. Too few (rerank top-10) and the right chunk may not survive stage 1 to be reranked at all; too many (rerank top-500) and you pay latency and compute for reordering chunks that were never going to make the cut. Teams tune it by measuring recall of stage 1 at various depths — a preview of the evaluation lesson.
Bi-encoder vs cross-encoder: where the dot product lives
The two architectures differ in exactly one structural decision: when the query and document are allowed to interact.
A bi-encoder computes score(q, d) = sim(E(q), E(d)) — two independent encoder passes, then a similarity (dot product or cosine) between the resulting vectors. Because E(d) does not depend on the query, it can be computed offline and indexed. The interaction between query and document is deferred to the very last operation, a single dot product between two fixed vectors. Everything the model wants to say about the document must survive compression into one Embedding before the question is known — which is precisely why subtle, query-specific distinctions get lost.
A cross-encoder computes score(q, d) = F(q ⊕ d) — one encoder pass over the concatenated pair, with a scoring head on top. Attention operates across the boundary from layer one: query tokens attend to document tokens and vice versa, so the representation of every document token is conditioned on the specific question being asked. Nothing is compressed prematurely — and nothing is reusable across queries. Sentence-BERT (Reimers & Gurevych, 2019) was born from exactly this tension: cross-encoders scored sentence pairs accurately but made large-scale comparison computationally impractical, so the authors trained bi-encoders whose independent embeddings could be compared cheaply — accepting an accuracy gap that two-stage systems later papered over by using both.
| Property | Bi-encoder (stage 1) | Cross-encoder (stage 2) |
|---|---|---|
Query–document interaction | Only at the final dot product between two fixed vectors | Full attention between query and document tokens, every layer |
Precomputable offline? | Yes — document vectors are query-independent | No — every score depends on the specific query |
Per-query cost | One query embedding + ANN index lookup | One full forward pass per candidate pair |
Scales to | Millions of chunks | Dozens of candidates |
Optimized for | Recall — is the right chunk in the net? | Precision — is the right chunk ranked first? |
Tool: Embedding Space Explorer — Feel the bi-encoder’s limits yourself: in the Embedding Explorer, compare pairs where single-vector similarity ranks a near-miss above the true answer — the gap reranking exists to close.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.