One-shot RAG vs agentic search
Lesson 2 of 5 in RAG for Agents: From One-Shot Retrieval to Agentic Search.
The classic RAG pipeline is one-shot: embed the user’s question, fetch top-k, stuff the chunks into the prompt, generate. One retrieval, no second chances. It is fast, cheap, predictable — and it quietly assumes something false: that the user’s question, as phrased, is a good search query for the text that answers it.
That assumption breaks three ways. Vocabulary mismatch: the question is phrased nothing like the answer (hybrid search softens this, but only so far). Multi-hop questions: ‘Did the vendor we chose for the Berlin office meet the security requirements from the 2025 audit?’ — no single chunk contains that answer; you need the audit requirements first, then the vendor decision, then a comparison. Underspecified questions: the right search terms only become obvious after you have seen some results.
Agentic search hands the retrieval loop to the agent. Instead of one fixed retrieval, the model formulates a query, inspects what comes back, reformulates based on what it learned, follows leads across hops, and — critically — decides when it has enough to answer. Search stops being a pipeline stage and becomes a tool call inside the agent loop, used as many times as the question demands. (This is also where MCP enters: exposing your corpus as MCP resources and search tools is the standard way to let any agent drive this loop over your data.)
The agentic search loop
- Question arrives
- Formulate search query
The agent writes a query — which need not resemble the user’s words. It can decompose a multi-hop question into its first hop.
- Run search (tool call)
Keyword, vector, or hybrid — the retrieval machinery from lesson 1, now invoked as a tool.
- Inspect results
The agent reads the hits: relevant? contradictory? pointing somewhere else? Each observation lands in context and informs the next move.
- Enough evidence to answer?
The agent’s own sufficiency judgment — the capability one-shot RAG simply does not have.
- Search budget left?
The runtime enforces a cap on search iterations — a stopping condition, exactly like any other agent loop.
- Reformulate / follow leads
Narrow terms, try synonyms the results revealed, or chase the next hop (‘the audit doc mentions Annex C — search for Annex C’).
- Answer with citations
- Answer with caveats — or say what could not be found
| Dimension | One-shot RAG | Agentic search |
|---|---|---|
Retrievals per question | Exactly one, using the user’s phrasing (or a single rewrite) | As many as the agent decides, each informed by prior results |
Multi-hop questions | Structurally cannot chain — the second hop’s query does not exist until the first hop’s answer is read | The native case: follow leads hop by hop |
Bad first query | Fatal — garbage in, confident garbage out | Recoverable — the agent sees weak results and reformulates |
Knows when it doesn’t know | No — it answers from whatever top-k returned, relevant or not | Partially — the sufficiency check lets it keep digging or admit defeat |
Cost and latency | One model call; predictable, cheap, fast | N model calls with growing context; expensive, slow, variable |
Debuggability | Simple: inspect the one query and the k chunks | Needs a full trace of every query, result set, and decision |
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.