A Worked Frame

Lesson 4 of 4 in Cost Modeling.

Here is the whole module as a procedure you can run this week. Keep the workload in symbols — the tools hold the live numbers.

The workload. R requests per month; each carries I input tokens (measure by tokenizing real traffic — template, retrieved context, history) and produces O output tokens, so T_in = R × I and T_out = R × O. Record three shape facts alongside the volumes: the peak-to-average ratio, the cache-hit fraction h on input prefixes, and the share s of traffic that is genuinely latency-sensitive.

The serverless path. C_api = R × [I(1 − h)p_in + I × h × p_cached + O × p_out], with the (1 − s) share re-priced at batch-tier rates where it qualifies. Every rate comes from the Token Cost Estimator.

The provisioned path. Convert peak demand into the platform’s capacity units — tokens per minute at peak, expressed in MUs, PTUs, or GSUs — and multiply by the hourly unit price for the term you would actually sign. This path prices like a fleet (units × hours), but the platform owns X and the ops.

The self-hosted path. Size the fleet from peak: N = peak tokens/sec ÷ X, with X measured for your engine and model via GPU Sizing. Then F = N × r × H, achieved u = V ÷ (N × X × 3600 × H), plus C_ops and the hidden lines from the previous lesson.

The costing procedure

Characterize the workload
Measure R, I, O from real traffic, tokenized with the target model’s own tokenizer. Record peak-to-average ratio, cache-hit fraction h, and latency-sensitive share s.
Price the serverless path
C_api = T_in(1 − h)p_in + T_in·h·p_cached + T_out·p_out, with the batchable share re-priced at discounted tiers. Live rates: the Token Cost Estimator.
Price the provisioned path
Convert peak tokens-per-minute into capacity units (MUs, PTUs, GSUs) × hourly unit price × term. Check the spillover story: what happens to requests above the reservation?
Price the self-hosted path
N = peak ÷ X with X measured (GPU Sizing tool), F = N·r·H, divided by achieved utilization for cost per served token. Add C_ops at loaded cost.
Add the hidden lines
Egress, storage (artifacts, logs, vector indexes), retry and over-generation multipliers, dev/staging environments, people.
One winner from V to 10×V?
Break-even is a curve, not a point. Rerun the comparison at current volume, expected growth, and an ambitious upside — note which assumptions each crossing depends on.
Model the hybrid
Reserved or self-hosted capacity for the steady baseline, per-token spillover above it — the pattern the platform docs themselves describe. Choose the split that maximizes utilization of the fixed layer.
Decide, rank sensitivities, revisit
Present the decision with its sensitivity ranking and an expiry date: rates, models, and traffic shapes all move, so the model is a living artifact, not a one-off memo.

A cost model earns trust through sensitivity, not precision. Before presenting any number, vary each input and watch which one moves the answer: in most chat-shaped workloads, output length O and utilization u dominate, because O multiplies the highest per-token rate and u divides the entire self-hosted side. The cache-hit fraction h matters enormously for agent-style traffic with long shared prefixes and barely at all for one-shot queries. The rates themselves — p_in, p_out, r — move in steps rather than drifts: a model swap, a tier change, a commitment term. That makes them decisions, not weather.

The encouraging part: the variables with the highest leverage are the ones you control. Capping O, structuring prompts for stable prefixes, and routing latency-tolerant work to discounted tiers are engineering changes with direct, formula-visible cost effects. That is the practical payoff of doing this in symbols — you know where a week of effort pays and where it rounds to zero.

Sensitivity ranking for a typical chat-shaped workload. Verify against your own numbers in the tools — the ranking shifts with traffic shape.
Variable	Where it enters	Leverage	Your lever
Output tokens `O`	Multiplies the highest rate on the serverless side; stretches serial decode time on the fleet side	High — output rates run a multiple of input on most verified price rows	Max-token caps, stop sequences, terser output formats
Utilization `u`	Divides the entire self-hosted (and provisioned) side	High — halving `u` doubles cost per served token	Hybrid baseline + spillover; consolidating workloads; scale-to-zero for non-prod
Cache hits `h`	Discounts the input share on the serverless side; raises effective `X` when self-hosting	Medium–high when prompts share long stable prefixes	Prefix-stable prompt design; cache-aware routing
Rates `p_in`, `p_out`, `r`	Scale their whole side of the inequality, in steps	Step changes — model choice, tier choice, commitment term	Model right-sizing; batch/Flex tiers; term commitments
Throughput `X`	Sets fleet size `N` for a given peak	High for self-hosting — engine quality is a price	Engine tuning, continuous batching, quantization

In production

Sooner or later finance asks what the LLM platform costs per unit of business value, and what it will cost at ten times the traffic. The team that already has the symbolic model answers in an afternoon; the team that does not reconstructs six months of invoices. Build it before they ask.

AWS

Model the Bedrock meters explicitly: on-demand tokens with Priority and Flex tiers around Standard, Provisioned Throughput in Model Units billed hourly on no-commitment, 1-month, or 6-month terms, and Marketplace models deployed to SageMaker AI-managed endpoints — dedicated compute, so cost them like a fleet, not like a meter. Two documented rules change your cost structure, not just the totals: a model customized in Bedrock must run on Provisioned Throughput, so a fine-tune converts you from usage billing to capacity billing; and cross-region inference through global profiles is documented at roughly 10% below standard pricing, with the routing itself at no additional cost.

Azure

Azure quota is denominated in tokens-per-minute per region, model, and deployment type — so the cost model and the capacity model are the same spreadsheet: the TPM you allocate to a deployment sets both its rate limit and its scaling ceiling. Provisioned deployments are purchased as PTUs guaranteeing a set processing capacity, with fungible provisioned throughput letting quota and reservations flex across supported models, while managed compute bills VM core-hours — three different meters inside one platform. Batch deployments draw on a separate enqueued-token quota, so routing latency-tolerant work there collects the discount without cannibalizing online throughput.

Google Cloud

Vertex documents the hybrid this module keeps arriving at: buy Provisioned Throughput in GSUs for the baseline (plans run from 1 week to 1 year) and let overage spill to Standard pay-as-you-go, controllable per request. Priority and Flex PayGo tiers price latency in both directions — a premium to jump the queue, a discount to wait — and Batch handles the asynchronous bulk from a dynamically allocated shared pool. Model the spillover explicitly: the reservation caps your fixed cost, the spillover keeps peak requests alive, and the split between them is precisely the utilization decision from lesson two.

Tool: Token Cost Estimator — Run the whole frame with live verified prices: put your R, I, and O into the Token Cost Estimator, compare models and tiers, and pair it with the GPU Sizing tool for the self-hosted side of the inequality.