RAG vs long context vs fine-tuning
Lesson 4 of 5 in RAG for Agents: From One-Shot Retrieval to Agentic Search.
There are three ways to get knowledge into a model’s answers, and every architecture review eventually litigates them. RAG: index the corpus, retrieve per query. Long context: skip the index and put the documents straight into the prompt. Fine-tuning: train the knowledge into the weights themselves.
Each one is periodically declared the winner. The truth is duller and more useful: they optimize different variables, and four honest criteria decide almost every case — freshness (how often the knowledge changes), corpus size (does it physically fit in a context window?), per-query cost (who pays, and when), and provenance (must the answer cite where it came from?).
| Criterion | RAG | Long context | Fine-tuning |
|---|---|---|---|
Freshness | Best — update the index, next query sees it | Good — swap the documents in the prompt | Worst — knowledge frozen at training; updates mean retraining |
Corpus size | Effectively unbounded — millions of documents | Hard-capped by the window; fine for a handful of documents, impossible for a wiki | Large corpora can go in, but recall is lossy and unverifiable |
Per-query cost | Low — retrieve a few chunks, small prompt | High — you pay for every document token on every query (prompt caching softens repeat hits) | Lowest at query time — but you paid up front in training runs and must repay on every update |
Provenance / citations | Native — you know exactly which chunks informed the answer | Possible — the model can cite from the prompt, and the sources are at least known | None — knowledge is smeared across weights; the model cannot tell you where a ‘fact’ came from |
What it’s actually for | Facts from a large, changing, auditable corpus | Deep reasoning over a small, known set of documents (a contract, a codebase slice, a case file) | Behavior, not facts — style, format, domain vocabulary, tool-use patterns |
Which knowledge strategy?
Interactive decision tree — outcomes:
- Fine-tune — you are shaping behavior, not storing facts
Fine-tuning excels at exactly this: consistent tone, strict output formats, domain vocabulary, learned tool-use patterns. Just don’t expect it to be a fact store — for knowledge, pair it with RAG.
- Long context — skip the index
A small, known document set with deep cross-document reasoning is long context’s home turf: no chunking losses, no retrieval misses, the model sees everything at once. You pay per token per query — acceptable at low volume, and prompt caching helps on repeats.
- RAG — retrieval as cost control
At high query volume, re-sending the same documents on every call burns money for no accuracy gain. Index once, retrieve the few chunks each query needs, and the per-query bill collapses.
- RAG — freshness or provenance leave no real alternative
A large, changing, or audit-bound corpus is RAG’s defining case: update the index and the next query sees it; every answer traces to the chunks that produced it. Fine-tuning is stale on arrival here, and the corpus will never fit a context window.
- Still RAG — fine-tuning is not a fact store
Even with a stable corpus and no citation requirement, training facts into weights gives you lossy, unverifiable recall — and every corpus update becomes a training run. RAG stays the default for knowledge; add fine-tuning only if behavior also needs shaping.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.