Why Sparse Models Exist

Lesson 1 of 3 in Mixture of Experts: Sparse by Design.

In a dense Transformer, every Parameter participates in every Token. That coupling is the quiet tax on scale: to make a Dense model smarter, you make it bigger — and every extra parameter adds compute to every single token you ever process, whether that token needed the extra capacity or not.

Mixture of Experts (MoE) breaks the coupling. Take the feed-forward network — the block component that holds most of a transformer’s parameters (with the usual 4× expansion, roughly two-thirds of each block’s weights live there) — and instead of one FFN, keep N of them side by side. Each copy is called an Expert. In front of the experts sits a tiny learned Router: for each token, at each MoE layer, it scores all N experts and sends the token’s hidden state through only the top k — commonly one or two. The rest of the experts do nothing for that token. That is Sparse activation: the model’s total parameter count grows with N, but the compute per token grows only with k.

The result is a model with two different sizes, and you need both to reason about it. Total parameters measure what the model knows and what your hardware must hold. Active parameters measure what each token actually pays for in compute. A dense model has one number; an MoE model deliberately splits them apart.

A vertical stack showing a token’s path through one MoE block: the token hidden state enters shared self-attention, then a small router layer, which selects Expert 2 and Expert 7 (highlighted) out of eight experts; the six unselected experts are marked as idle, and the two expert outputs are combined by router weight and added back to the residual stream.

One MoE transformer block, bottom-up, for a single token. Attention is shared by all tokens as usual; the dense FFN is replaced by a bank of eight experts, of which the router activates only two. Toy example — real models vary N, k, and which layers are MoE. (illustrative — source: After Shazeer et al. (2017), arXiv:1701.06538)

The idea is older than the transformer. Shazeer et al. (2017) put a sparsely-gated MoE layer between LSTM layers and trained language models with up to 137 billion parameters — at a time when that count was so absurd the paper is literally titled Outrageously Large Neural Networks. The Transformer era simplified and scaled the recipe: Switch Transformer (Fedus, Zoph & Shazeer, 2021) showed that routing each token to just one expert (k = 1) works, and used it to train models past a trillion parameters. Open-weight models such as Mixtral brought the pattern into mainstream LLM serving.

One caution about the name: experts suggests tidy specialists — a biology expert, a legal expert. Published routing analyses say otherwise. The Mixtral authors, for example, report little evidence of topic-based specialization; expert choice appears to track token-level and syntactic patterns more than subject matter. Treat MoE as a capacity trick the optimizer exploits however it likes, not as a committee of recognizable specialists.

Key terms: Mixture of experts (MoE), Expert, Router, Sparse activation, Dense model, Feed-forward network (FFN)

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.