The Cost Shapes
Lesson 2 of 3 in In Production: Choosing Your Lever.
Every lever has two numbers: what it costs to build and what it costs to run. The totals matter less than the shapes — where the money sits and what multiplies it.
Prompting: near-zero build, per-call rent. Writing instructions and picking Few-shot examples costs engineering hours, nothing more. But everything you wrote is input tokens, resent on every call: a 2,000-token preamble on a million calls a day is two billion input tokens of rent a day, forever, growing in lockstep with traffic. The lever with the cheapest entry has the most relentless meter.
RAG: pipeline build plus index ops, per-query retrieval cost. The build is real engineering — Chunking strategy, an Embedding model, a Vector database or index, ingest and freshness pipelines. Then two run costs: the index itself is standing infrastructure that costs money while idle, and each query pays for retrieval — embedding the query, Approximate nearest neighbor (ANN) search, perhaps a Cross-encoder rerank — plus the retrieved passages entering the prompt as input tokens.
Fine-tuning: training build plus artifact ops, cheaper serving. The build is the biggest of the three: curating data, running training, and evaluating the result. The payoff is run-cost shape: behavior moves into the weights, so the per-call prompt shrinks — or a smaller model becomes sufficient for the task. In exchange you now operate a model artifact: hosting it, versioning it, and re-testing it whenever anything underneath moves.
Bar chart with six bars comparing illustrative build and run costs for three adaptation levers. Prompting: build 1 unit, run 8 units. RAG: build 5 units, run 5 units. Fine-tuning: build 9 units, run 3 units. The bars show that prompting concentrates cost in running, fine-tuning concentrates cost in building, and RAG spreads cost across both.
Two multipliers decide which shape wins.
Traffic multiplies run cost. At two hundred calls a day, the few-shot preamble’s rent is a rounding error and the fine-tune’s build cost never pays for itself — prompting wins almost by default. At millions of calls a day the inequality flips: rent dominates everything, and the lever with the expensive build and the cheap meter starts to amortize. Same task, same model, opposite answer — traffic is the variable people forget to state when they argue about levers.
Change rate multiplies build cost. Every time the ground truth moves, you pay a slice of build again. RAG absorbs change gracefully — an index refresh is an ingest job. A fine-tune absorbs change expensively — retraining, re-evaluating, redeploying. Fast-changing knowledge plus weights is the worst pairing on the board; stable behavior plus weights is the best.
Break-even mechanics, one level down
Caching bends the prompting line. Prefix caching discounts the stable front of a prompt: if your preamble never changes and sits first, repeated calls pay a reduced rate for those tokens instead of full Prefill price. That moves prompting’s break-even point — a well-structured static preamble rents for less than its raw token count suggests, which is one more reason context structure (static first, volatile last) is an economic decision, not a stylistic one.
RAG’s per-query cost decomposes unevenly. Embedding the query is a small model call — cheap. Approximate nearest neighbor (ANN) search over an HNSW-style index is milliseconds of compute — cheap per query, but the index underneath is provisioned infrastructure, a cost that exists at zero traffic. A Cross-encoder rerank scores each candidate with a full forward pass — the expensive stage, which is why rerankers run over dozens of candidates, not thousands. And the dominant line is usually none of these: it’s the retrieved passages themselves, entering the LLM as input tokens on every query. Retrieval depth is a token-budget decision.
Fine-tune ops are recurring, not one-time. The artifact binds to a base model version: when the provider deprecates that base, retraining is forced on someone else’s schedule. If the task drifts, the training data and eval suite need maintenance — the pipeline is the asset, the artifact is a snapshot. LoRA changes the equation at both ends: training touches a small Adapter instead of every weight, and multi-adapter serving lets one hosted base model serve many adapters, so the hosting floor is shared across use cases instead of paid per task.
The honest comparison is lifetime cost at expected traffic — build plus run-rate times months, for each lever. Both terms are estimates, and traffic estimates are usually wrong. That uncertainty is itself an argument for the ladder: levers that fail cheap are worth a quality discount.
In production
Each cost shape lands on a different part of the cloud bill, metered differently — which is why two teams with the same monthly total can be in wildly different positions.
AWS
On Amazon Bedrock the prompting rent is the input-token meter, and prompt caching is the discount mechanism for stable prefixes — the billing structure literally rewards putting static content first. The RAG shape splits across services: embedding-model invocations at ingest and query time, plus a vector store (an OpenSearch-style index or other backing store) that bills as standing infrastructure whether or not anyone queries it. The fine-tune shape is a customization job at build time, then hosting for the custom artifact — capacity that costs while idle. The run-cost saving is real only when cheaper traffic outweighs that hosting floor.
Azure
Azure AI Foundry makes the rent visible twice: input tokens hit both the bill and the tokens-per-minute quota, so a bloated preamble burns throughput headroom as well as money. Azure AI Search indexes are sized and billed as standing capacity — the idle-index cost in its clearest form. Fine-tuned deployments carry a hosting charge separate from token billing. The pattern to read from any Azure invoice: build costs appear as jobs, run costs as meters, and indexes as meters that never stop.
Google Cloud
Vertex AI’s count-tokens API is the pre-flight tool for the rent conversation — measure exactly what your preamble and retrieved passages cost before shipping a template change. Vector search serving is provisioned capacity (the standing-index shape again), tuning runs as batch jobs, and tuned models deploy to endpoints with their own serving costs. Across all three clouds the taxonomy is identical: token meters, standing indexes, and hosted artifacts — learn to see every architecture diagram as a sum of those three.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.