Inference & Serving

Sampling, the KV cache, quantization, batching — where latency and cost are made

• Prefill and Decode — One request, two very different phases: the parallel prefill and the serial decode — the split behind every latency number and price sheet in this field. (3 lessons, 40 min)
• Decoding and Sampling — Greedy, beam, temperature, top-k, top-p: how one token actually gets chosen, and how to pick settings you can defend. (4 lessons, 45 min)
• The KV Cache — The memory that makes generation affordable: what gets cached, what it costs, paged attention, and prefix reuse. (4 lessons, 45 min)
• Context Windows and Long Context — The window as a budget: what long context really costs, the techniques that stretch it, and why the middle of a long prompt is a bad neighborhood. (4 lessons, 40 min)
• Quantization — Fewer bits, same model — mostly: what gets quantized, the methods that made it routine, and how to check what it cost you. (4 lessons, 45 min)
• Speculative Decoding — Draft cheap, verify exact: how a small model makes a big one faster without changing a single output token. (3 lessons, 40 min)
• Continuous Batching and Serving Engines — Why a GPU serving one request at a time is a tragedy, how continuous batching fixes it, and a map of the engines that do this for a living. (4 lessons, 45 min)
• In Production: TTFT, TPOT, and the Metrics That Matter — The domain’s production capstone: the latency vocabulary, the throughput–latency frontier, and how serving choices surface on the three clouds. (3 lessons, 40 min)