Break-Even Thinking
Lesson 2 of 4 in Cost Modeling.
Strip away the vendor names and the build-versus-buy question is one inequality. Define the workload and the two paths in symbols:
T_in,T_out— monthly input and output tokens, measured by tokenizing real traffic, never guessedp_in,p_out— per-token rates for the model you would call (live numbers belong in the Token Cost Estimator, not in prose)N— GPUs in the self-hosted fleet;r— hourly rate per GPU;H— hours per month (≈ 730)X— effective Throughput per GPU in tokens per second, under your engine on your traffic (size it with GPU Sizing)u— Utilization: the fraction of the fleet’s paid capacity that serves real traffic
The serverless path costs what you use: C_api = T_in × p_in + T_out × p_out. The self-hosted path costs what you run: F = N × r × H. The naive break-even sets them equal — with total volume V = T_in + T_out and blended rate p_blend = C_api ÷ V, the fleet wins once V > V* = F ÷ p_blend.
Naive, because F does not serve V automatically. The fleet can serve at most N × X × 3600 × H tokens a month — and in practice u times that.
Line chart with three series plotted against monthly token volume in units of the break-even volume V-star. A diagonal line through the origin represents serverless cost, rising linearly with volume. A horizontal line at height 1 represents a fixed self-hosted fleet at full utilization; it crosses the diagonal at volume 1. A second horizontal line at height 2 represents the same workload when utilization halves, requiring twice the fleet; it crosses the diagonal at volume 2, showing that lower utilization pushes break-even further out.
Why is u so hard to keep high? Because you size for peak and pay for always. Interactive products sleep sixteen hours a day; peaks run multiples of the average; and the fleet must hold enough capacity for the worst minute of the week. Autoscaling closes less of the gap than it does for stateless web servers — model weights take minutes to load, so scale-up lags spikes and scale-to-zero buys idle savings at the price of cold starts measured in minutes. Meanwhile capacity reservations and Quota realities (a theme of GPUs and Accelerators) push the other way: released GPU capacity is capacity you may not get back, so fleets tend to hold their peaks. Spot capacity cuts the hourly rate r in exchange for interruptions — useful for batch and eval traffic, dangerous for the latency-sensitive path.
The managed platforms sell the resolution directly: cover the steady baseline with provisioned capacity, and let spillover ride the per-token meter. Vertex’s consumption docs describe exactly this pattern — Provisioned Throughput for the baseline with overage billed as standard pay-as-you-go — and you can reproduce the same shape on any of the three, or between a self-hosted fleet and a serverless fallback.
The full inequality, with caching and batching corrections
The two-line version prices the standard tiers only. A production cost model adds three corrections, each with a named knob.
Caching corrects the serverless side. With cache-hit fraction h over input tokens and a discounted cached-input rate p_cached (where the provider offers one — our price table deliberately models the standard tier only, so let the estimator carry current numbers):
C_api = T_in × (1 − h) × p_in + T_in × h × p_cached + T_out × p_out
Prefix caching rewards prompt structure: a stable system-prompt-and-tools prefix ahead of the variable suffix turns h from an accident into a design parameter.
Batch tiers correct the serverless side again. Split volume into a latency-sensitive share s at standard rates and a batchable share (1 − s) at discounted asynchronous rates. All three platforms document lower-cost tiers for latency-tolerant work — Bedrock’s Flex tier, Azure’s Batch deployment types, Vertex’s Flex and Batch. Work that can wait should never pay the interactive rate.
Engine quality corrects the self-hosted side. X is not a hardware constant — it is what your Serving engine achieves on your traffic. Continuous batching raises it dramatically over one-request-at-a-time serving; Quantization trades precision for throughput and memory headroom; and self-hosted prefix caching raises effective X on cache-friendly traffic instead of lowering a rate. Every improvement to X shrinks the fleet a given peak requires — moving V* without any price changing.
Put together, the fleet wins when
r ÷ (u × X × 3600) < [T_in(1 − h)p_in + T_in × h × p_cached + T_out × p_out] ÷ (T_in + T_out)
— with every symbol on both sides measured or quoted today, never assumed. The inequality is stable; the numbers in it are not.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.