Latency engineering for LLM workloads
Lesson 1 of 3 in Performance and Monitoring for GenAI.
LLM latency has an anatomy, and optimizing without it means pulling the wrong lever. The wait splits into time to first token (TTFT — network, queueing, retrieval, and the model processing your whole input) and generation time (every output token, one by one). CloudWatch separates them for Bedrock: TimeToFirstToken for the streaming APIs, and InvocationLatency measured to the last token. The split matters because users feel TTFT, not totals — a response that starts in 400 ms and streams for six seconds feels fast; the same six and a half seconds delivered silently feels broken.
One derived number closes the diagnostic loop: output tokens per second (OTPS) — metric math of OutputTokenCount / (InvocationLatency − TimeToFirstToken) × 1000. When latency climbs, OTPS tells you whether the model got slower (OTPS fell) or just chattier (OTPS flat, output count up) — two different problems with two different fixes.
Horizontal bar chart of the illustrative share of total latency for a single streamed retrieval-augmented response: token generation (decode) 65%, prefill to first token 18%, retrieval including embedding and vector search 12%, network and queueing overhead 5%. Hints name the lever per slice: cap and tighten outputs or pick a faster model for decode; prompt caching and shorter prefixes for prefill; index tuning and parallel retrieval for the retrieval hop; connection reuse and Region proximity for overhead. Relative weights for one illustrative workload, not benchmark data.
Stream the response
The highest-leverage UX move: ConverseStream / InvokeModelWithResponseStream deliver tokens as they generate, so perceived latency collapses to TTFT. Total latency is unchanged — the lever is when the user starts reading. The implementation lives in streaming responses; here it is the first answer to "feels slow".
Latency-optimized inference
For time-sensitive paths, latency-optimized inference serves select models on a faster service tier: set performanceConfig: { "latency": "optimized" } on the runtime call. Verified October 2026 (feature in preview): offered for Amazon Nova Pro, Anthropic Claude 3.5 Haiku, and Meta Llama 3.1 70B/405B through cross-Region inference. Past the feature's usage quota, requests fall back to standard and bill at standard rates — the served tier is visible in the API response and CloudTrail.
Parallelize independent work
Complex workflows serialize by habit: five independent subtasks × 2 s each = 10 s that could be ~2 s of concurrent model invocations. Fan out what is independent (per-document extraction, multi-aspect analysis), gather, then synthesize. The budget you spend is quota headroom — watch InvocationThrottles and apply backoff, because a parallel burst is indistinguishable from a traffic spike.
Pre-compute the predictable
The fastest inference is the one that already happened. Queries you can enumerate — daily account summaries, dashboard narratives, FAQ answers — generate ahead of demand (batch inference on a schedule, results into DynamoDB or a cache) and serve as a lookup. Pre-computation converts user-facing latency into off-peak, discounted batch spend.
Tune the retrieval hop
In RAG, retrieval sits inside TTFT. Index optimization (right algorithm parameters, sharding, warm replicas), query preprocessing, and hybrid search — lexical + semantic combined with custom scoring, with a Reranker where quality demands it — cut both retrieval latency and wasted context. Depth lives in embeddings and hybrid search; the operations angle (monitoring the vector store) returns in lesson 3.
Sampling dials: temperature, top-k, top-p
The AIP-C01 exam guide wants parameter judgment, not definitions. Per-token sampling cost barely moves, but the dials shape latency and quality indirectly: low temperature / tight top-k → focused, shorter, reproducible outputs — better for extraction, structured tasks, golden-dataset comparisons, and cache hit rates; higher temperature / top-p → diversity for creative work at the price of verbosity (more output tokens = more time) and run-to-run variance. Requirements decide: "consistent, auditable" points low; "varied, exploratory" points high.
Key terms: Time to first token (TTFT), Latency-optimized inference
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.