In Production: TTFT, TPOT, and the Metrics That Matter

The domain’s production capstone: the latency vocabulary, the throughput–latency frontier, and how serving choices surface on the three clouds.

Content last verified 2026-09.

Lessons

1. The Latency Vocabulary
2. The Throughput–Latency Frontier
3. The Cloud Serving Menu

Sources

• Kwon et al. (2023) — Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM)
• Pope et al. (2022) — Efficiently Scaling Transformer Inference
• Yu et al. (2022) — Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022)