What MoE Costs You
Lesson 3 of 3 in Mixture of Experts: Sparse by Design.
Every MoE spec sheet gives you two numbers, and confusing them is the single most common MoE mistake in practice. Total parameters are what your hardware must hold: every expert, at every MoE layer, has to sit in accelerator memory, because the router may send the very next token to any of them. Active parameters are what one token’s forward pass actually computes with: the shared layers plus the k selected experts. Compute cost tracks active parameters; the memory footprint — and much of the weight-loading bill — tracks total parameters.
Quality, roughly speaking, benefits from the total; your per-token FLOPs are set by the active. That gap is the entire commercial appeal of MoE — and the entire operational headache.
Three horizontal bars. Dense model, parameters computed per token: 100%. MoE, parameters held in memory: 100%. MoE, parameters computed per token: about 28%. The gap between the last two bars illustrates that an MoE holds far more parameters than any single token uses.
The third cost is subtler: batching gets complicated. A dense model processes a batch with the same big matrix multiplies for every token — GPUs love it. In an MoE, each token in the batch picks its own experts, so the serving stack must group tokens by expert, run N smaller matrix multiplies of unpredictable sizes, and scatter the results back. Expert loads are uneven and shift with the input distribution, so some experts become stragglers while others idle. Spread experts across multiple devices (expert parallelism) and every MoE layer adds all-to-all communication — tokens physically travel to their experts and back. None of this changes the KV cache, which belongs to the shared attention layers — but it makes MoE throughput far more sensitive to batch size and traffic mix than a dense model of equal active size.
| Dimension | Dense model | MoE model (top-k of N experts) |
|---|---|---|
Compute per token | Proportional to its one parameter count | Proportional to active parameters — shared layers + k experts |
Accelerator memory for weights | Same number again — memory and compute move together | Proportional to total parameters — several times the active count |
Quality lever | Grow the model → every token gets more expensive | Grow N (total capacity) while holding k → per-token compute barely moves |
Batching behavior | Uniform, predictable matrix multiplies | Tokens fan out to different experts; uneven loads, all-to-all traffic when experts are sharded |
Training pitfalls | The usual ones | All the usual ones plus expert collapse, balancing losses, capacity tuning |
In production
MoE trades FLOPs for VRAM: you pay for total parameters in memory but only active parameters in compute. Whether that trade favors you depends almost entirely on who owns the memory problem — you, or a managed API provider.
AWS
Self-hosting an open MoE on GPU instances means provisioning accelerator memory for the total parameter count — a model whose per-token compute would fit on a modest setup can still demand multi-GPU tensor or expert sharding just to load, and sharded experts add all-to-all traffic that makes interconnect bandwidth part of your latency budget. Behind a managed endpoint such as Bedrock the pricing is per token, so the provider absorbs the memory footprint and batching complexity; compare candidates on quality per dollar at your traffic shape, not on parameter counts.
Azure
The same fork applies between the model catalog’s hosted endpoints and running your own GPU VMs or AKS pools. Hosted MoE endpoints hide the footprint entirely. Self-managed, plan capacity from two ceilings, not one: memory sized to total parameters and compute sized to active parameters — and remember MoE throughput is batch-sensitive, so a deployment that benchmarks well under saturated load can disappoint on spiky, low-concurrency traffic where dense models of equal active size are more forgiving.
Google Cloud
On self-managed GPU or TPU capacity, MoE rewards exactly what large accelerator pods provide — big pooled memory and fast interconnect for expert parallelism — which is one reason sparse models have long been trained on such systems; at small self-hosted scale those advantages invert into cost. Managed endpoints on Vertex AI charge per token, making the sparse/dense distinction invisible to you operationally: it survives only as the quality-versus-price curve you evaluate against.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.