Inside the Transformer

Tokens, embeddings, attention, and the block — the machine, opened up

• Tokenization: Text Becomes Tokens — Why models read subword tokens instead of words, how byte-pair encoding builds a vocabulary, and the artifacts every engineer eventually hits. (4 lessons, 45 min)
• Embeddings: Tokens Become Vectors — The embedding matrix, what geometry in representation space means, contextual versus static embeddings, and the unembedding back to logits. (4 lessons, 40 min)
• Positional Encoding: Teaching Order to a Set — Attention is order-blind by construction. Sinusoidal and learned positions, RoPE, and how position schemes shape context extension. (4 lessons, 40 min)
• Self-Attention: Every Token Looks at Every Token — Queries, keys, and values; the scaled dot-product equation term by term; causal masking — the mechanism that made the transformer. (4 lessons, 50 min)
• Multi-Head, Multi-Query, Grouped-Query Attention — Why many small heads beat one big one, what heads actually learn, and how MQA/GQA trade attention quality for serving memory. (3 lessons, 40 min)
• Inside the Block: FFN, Residuals, Normalization — The other half of the transformer block: feed-forward networks, the residual stream, and why normalization placement matters. (4 lessons, 45 min)
• Mixture of Experts: Sparse by Design — Routers, experts, and load balancing — how MoE models activate a fraction of their parameters per token, and what that costs in memory. (3 lessons, 40 min)
• One Token's Journey: The Full Forward Pass — The whole stack assembled end to end — from input text to next-token probabilities, and where the parameters actually live. (4 lessons, 45 min)
• Architecture in Production: Design Choices Set Your Bill — The domain's production capstone: how attention's quadratic tax, GQA, MoE, and context length decisions surface as cost, latency, and capacity on real cloud hardware. (3 lessons, 40 min)