Architecture Choices Meet the Bill
Lesson 1 of 3 in Architecture in Production: Design Choices Set Your Bill.
Every module in this domain ended with the same quiet warning: this choice will cost you. This capstone collects the debts. A model’s architecture summary — Vocabulary size, width, depth, head layout, Expert count, context limit — reads like an engineering datasheet, but for anyone who runs the model it is really a bill forecast. Each number sets one or more meters spinning, and the meters have names your finance team already knows: accelerator memory, compute, latency, dollars.
The trick to reading a spec sheet this way is that there are only two fundamental meters. The first is bytes — how much data must sit in accelerator memory (the Weights, the KV cache, activations) and how much must move through the memory system per Token. The second is FLOPs — the arithmetic each token’s forward pass performs. A standard estimate puts a forward pass at roughly 2 FLOPs per parameter per token — counting active parameters for MoE models — plus an attention term that grows with context (Kaplan et al., 2020; the exact accounting is the next lesson’s deep dive). Everything else you will ever be billed for — time to first token, tokens per second, sessions per GPU — is these two meters filtered through hardware limits and provider pricing.
| Design choice | Weight memory | KV cache per sequence | FLOPs per token | Where you feel it |
|---|---|---|---|---|
Vocabulary size (V) | Embedding (and unembedding, if untied) grow with V × d_model | No direct effect | The final logits matmul grows with V | Often runs backwards: better coverage → fewer tokens per text → every downstream meter, including billed tokens, shrinks |
Width (d_model) | Grows roughly with d² per layer | Grows with the cached vector width (≈ heads × head_dim) unless KV heads are cut | Grows roughly with d² | The blunt lever — quality, memory, and compute all move together |
Depth (layer count) | Linear in layers | Linear in layers — every layer keeps its own K/V | Linear in layers | Also latency: layers run one after another, so depth is sequential time |
KV heads (MQA / GQA) | Slightly smaller K and V projections | Shrinks proportionally — the whole point | Barely moves — scores are still computed for every query head | More concurrent sessions per accelerator; faster decode (less cache to re-read per step) |
MoE (N experts, top-k) | Grows with total parameters — idle experts stay resident | Unchanged — attention stays shared and dense | Grows with active parameters only | The FLOPs-for-VRAM trade, plus batching complexity and interconnect traffic when experts shard |
Context length (as used) | None | Linear in tokens actually held | The attention term grows with context — quadratically over a full prompt | Slow first token on long prompts; long-context surcharges; concurrency ceilings |
Two rows deserve a closer look, because they are the ones engineers most often misread.
Vocabulary size looks free — “it’s just the tokenizer.” It is not. The Embedding matrix holds V × d_model weights, the Unembedding (when untied) holds the same again, and every generated token pays a Logits matmul across the full vocabulary. Yet the effect that usually dominates in production runs the other way: a vocabulary that covers your languages and domains well emits fewer tokens for the same text, and every downstream meter — KV cache held, FLOPs spent, context consumed, tokens billed — counts tokens. A bigger, better-fitting vocabulary often lowers the bill even though it makes the model itself larger.
Depth versus width move the same meters at different exchange rates. For the usual block shape, the big matrices have d_model on both sides — roughly 12 × d² weights per block between attention and the FFN — so widening grows memory and compute quadratically, while deepening grows them linearly. But layers execute one after another: depth is sequential time that parallel hardware cannot fully hide. Two models of equal Parameter count can feel different to users partly because the deeper one pays more latency per token.
In production
Cloud pricing looks like a thicket of SKUs, but under every line item are the same two meters: bytes held or moved in accelerator memory, and FLOPs per token. Architecture decides how fast each meter spins.
AWS
Self-hosting on GPU instances, the spec sheet becomes a sizing checklist in meter order: do the weights fit in the accelerator pool at your serving precision; how much memory remains for KV cache — because that remainder, not requests per second, caps concurrent sessions; and do per-token FLOPs meet your latency target. Behind a managed per-token API such as Bedrock the meters do not disappear — they return as the shape of the price sheet: separate input and output rates, context-length limits, and throughput quotas are the provider metering bytes and FLOPs on your behalf.
Azure
Provisioned-throughput offerings ration exactly what the two meters measure: a capacity unit is, mechanically, a slice of accelerator memory, memory bandwidth, and compute. That is why two workloads with identical request counts can burn wildly different amounts of capacity — one moves more bytes per request (long contexts), the other more FLOPs (long generations). Plan reservations from token volumes and context shape, never from request counts.
Google Cloud
The meters also pick your hardware. Accelerator families differ in their ratio of memory capacity to memory bandwidth to raw FLOPs — and in interconnect, if you shard. A memory-hungry architecture (MoE totals, long-context caches) wants capacity and fast interconnect; a compute-hungry one (large dense models, prompt-heavy traffic) wants FLOPs. On managed Vertex AI endpoints the same trade arrives pre-made, as each model’s price and latency curve.
Key terms: d_model (model dimension), KV cache, Grouped-query attention (GQA), Mixture of experts (MoE), Embedding matrix, Unembedding
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.