The Decision Framework
Lesson 1 of 3 in In Production: Choosing Your Lever.
You now own every lever in this domain mechanically: prompting and Few-shot examples, Context engineering, the full Retrieval-augmented generation (RAG) pipeline from Chunking to Reranking, Fine-tuning with LoRA and QLoRA, and Distillation. The capstone question is the one production actually asks: which lever, for this problem, right now?
The answer is a ladder, climbed from the bottom:
- Context — change what you send: instructions, examples, structure.
- Retrieval — change what the model can look at: an index of your documents, searched per query.
- Weights — change the model itself: adapters or full fine-tuning.
One rule governs the climb: start at the cheapest, most reversible rung, and escalate only when a rung measurably fails. A prompt experiment costs an afternoon and rolls back with a git revert. A retrieval pipeline costs weeks and rolls back by repointing a data source. A fine-tune costs a dataset, a training run, and an evaluation suite — and leaves you operating a model artifact with a lifecycle of its own.
Before touching any lever, name the problem. Almost every adaptation request is one of three:
Knowledge problems. The model doesn’t know something: your private documents, anything past its Knowledge cutoff, facts too niche for pretraining to have absorbed. The tell is confident wrongness about your domain — Hallucination where the truth simply wasn’t available. Knowledge problems go to retrieval, because facts change faster than weights should, and because retrieved passages give you something a fine-tune never will: a citation you can audit — the mechanics of Grounding.
Behavior problems. The model knows enough but answers in the wrong shape: broken output format, off-brand tone, ignored policy, the wrong level of detail. Behavior problems go to examples first — In-context learning is cheap and instantly reversible — and to weights when the per-call budget of instructions and examples stops paying, or the pattern is too subtle to demonstrate in a handful of shots.
Cost problems. The answers are right but too slow or too expensive at your traffic. These are usually serving problems before they are adaptation problems — Quantization, caching, batching, and the rest of the serving menu covered in the Inference domain. Adaptation’s contribution comes after: Distillation or a fine-tuned smaller model that makes cheap capacity sufficient.
| Problem type | What you observe | First lever | Why it fits |
|---|---|---|---|
Knowledge | Wrong, stale, or missing facts; confident specifics about your domain that are simply false | Retrieval (RAG) — index your documents, search per query | Facts live in an index you can update in minutes, not in weights you retrain — and every answer carries passages you can audit |
Behavior | Right facts, wrong shape: broken format, off-tone, ignored policy | Instructions + few-shot examples; escalate to a fine-tune or adapter when the example budget stops paying | Behavior is a pattern, and patterns can be demonstrated in context first; weights are for patterns too broad or too subtle to re-demonstrate on every call |
Cost / latency | Answers are fine; the bill or the wait is not | Serving levers first (caching, quantization, batching); then distillation or a smaller fine-tuned model | Most cost problems are made — and fixed — at serving time; adaptation’s move is making a cheaper model good enough |
Capability ceiling | The model fails the task even with perfect context, perfect examples, and the facts in front of it | A different model — usually bigger, or a reasoning model | No adaptation lever adds reasoning that isn’t there; switching models beats fighting one |
Why insist on cheapest-reversible first, even when you suspect you’ll end up at weights anyway? Because every rung you skip is information you don’t have. Trying the prompt first answers the question that shapes everything downstream: can the model do this task at all when the facts and the format are in front of it? If yes, you have a knowledge or cost problem, not a capability one. If it fails on facts, retrieval; if it fails on form even with the facts present, behavior. The failed cheap experiment is the diagnosis — and it cost you an afternoon.
Reversibility is the other half. Production systems change: the model behind your API gets deprecated, your product ships features, your traffic doubles. A lever you can un-pull in minutes is worth more than its quality delta suggests, because it keeps every future decision open. Where a retrieval step is invoked by an agent that decides when to search, that orchestration is agent-scaffold territory — our sister AI Agent Academy covers it; the lever logic here is unchanged.
Key terms: Context engineering, Retrieval-augmented generation (RAG), Fine-tuning, LoRA, Grounding
Tool: Prompt vs RAG vs Fine-tune — Walk the framework interactively: the Adaptation Advisor asks the diagnostic questions in order, recommends a lever, and shows which of your answers would flip the recommendation.
In production
The ladder is visible in what the managed platforms sell: each rung is a product surface, and the order of the rungs matches the order you should try them.
AWS
Amazon Bedrock’s shape mirrors the ladder. The context rung needs nothing beyond the invoke API — system prompts and templates you version yourself, with prompt-management features to keep them out of application code. The retrieval rung is Knowledge Bases: managed ingest, chunking, embedding, vector storage, and a retrieve-and-generate call, so a knowledge problem can be attacked before you build a pipeline of your own. The weights rung is a model customization job that produces a new artifact you must host, version, and re-test. Notice the gradient: every rung upward adds an artifact you operate.
Azure
Azure AI Foundry exposes the same three rungs: prompt and configuration assets versioned inside the project; grounding on your own indexes through Azure AI Search — the “on your data” pattern — for the retrieval rung; and fine-tuning that produces a deployment with its own lifecycle, separate from the base model’s. The mechanism to internalize is burden transfer: the further down the ladder you reach, the more of the operational responsibility moves from the platform’s side of the line to yours.
Google Cloud
Vertex AI walks the same staircase: prompt design and management in the studio, grounding services and a managed RAG offering for the retrieval rung, and tuning jobs that produce tuned-model endpoints at the weights rung. Managed offerings compress the build cost of each rung — ingest pipelines and training loops you don’t write — but they do not change the run-cost shape of the lever you chose. That shape is the next lesson.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.