The Three Meters
Lesson 1 of 4 in Cost Modeling.
The cloud domain is full of architectural choices; nearly all of them are secretly this one. Every dollar an LLM workload spends reaches the invoice through one of three meters:
- Per-token. Serverless model APIs meter tokens in and tokens out, priced per million. No traffic, no charge. This is the default consumption mode on Amazon Bedrock, on Azure AI Foundry’s serverless deployments, and on Vertex AI’s pay-as-you-go tiers.
- Per capacity-unit-hour. Provisioned throughput sells you a reserved slice of a model’s serving capacity — defined in tokens per minute — billed by the hour for the length of a term, used or not.
- Per GPU-hour. Self-hosting rents the hardware itself: the model never appears on the invoice. You pay for instances by the hour, bring your own Serving engine, and own everything that happens on top.
A meter is not just a price — it is an incentive structure. Each one rewards a different traffic shape and punishes a different failure mode, and most real cost surprises are a workload sitting on the wrong meter for its shape.
Key terms: Provisioned throughput, Utilization, Quota, Capacity reservation, Egress
| Meter | You pay for | What it rewards | What it punishes |
|---|---|---|---|
Per-token (serverless) | Input and output tokens, metered per request. Idle time costs nothing. | Spiky, unpredictable, or low traffic; prototypes and experiments that might be turned off next quarter; zero capacity planning. | High steady volume — every token bills at list rate forever — and chatty outputs, since output rates exceed input rates on nearly every row of our verified price sample. |
Per capacity-unit-hour (provisioned) | A reserved throughput level — Bedrock MUs, Azure PTUs, Vertex GSUs — billed hourly for the term, whether traffic arrives or not. | Steady, latency-sensitive baselines that actually fill the reserved throughput; invoices finance can predict; guaranteed capacity when on-demand Quota would throttle you. | Idle reservations — the meter runs at 3 a.m. — and traffic that outgrows the purchased units mid-term or never grows into them. |
Per GPU-hour (self-hosted) | Instances or Kubernetes nodes by the hour — or a Managed endpoint on dedicated VMs (Azure managed compute bills VM core-hours; Vertex self-deployed models run on compute in your project). | Sustained volume at high Utilization; custom models and engines the managed catalogs will not serve; full control of data paths and hardware. | Low utilization above all — an idle GPU bills exactly like a busy one — plus the unmetered people-time this module ends on. |
Two refinements matter before any arithmetic.
Input and output are different meters inside the meter. Per-token pricing bills input and output separately, and output is the expensive side: on nearly every row of our verified price sample, the output rate is a multiple of the input rate. The mechanics live in Prefill and Decode — Prefill processes your whole prompt in one parallel pass, while Decode generates output one token at a time, occupying the accelerator for the entire generation. The billing asymmetry is that mechanical asymmetry passed through to you. A workload’s input/output mix is therefore the first number a cost model needs; compare real shapes in the Token Cost Estimator rather than trusting any single blended rate.
The per-token meter itself now has tiers. All three platforms document premium and discounted variants of pay-per-token: Bedrock lists Standard, Priority (fastest responses at a premium), and Flex (lower-cost for non-time-sensitive work) request tiers; Vertex documents the same Standard/Priority/Flex trio plus Batch for high-volume asynchronous jobs; Azure’s serverless deployment types split by routing scope — Global, Data Zone, regional — with Batch variants alongside. Latency tolerance is a discount you can choose to collect.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.