Scaling for GenAI traffic patterns
Lesson 2 of 3 in Performance and Monitoring for GenAI.
GenAI traffic breaks the assumptions classic auto scaling was built on. Requests are long and wildly variable — a streamed answer can hold a connection for 45 seconds — so request-per-second targets mislead. The backend is token-metered or GPU-bound, so CPU utilization is noise. And demand is bursty in human rhythms: the nine-o'clock surge, the lunch dip, the product-launch spike. Capacity planning therefore happens in GenAI's own units: tokens per minute and concurrent requests, not instances ("capacity planning for token processing requirements", in the exam guide's words).
Two worlds scale differently. On serverless Bedrock, there is no fleet to grow: your ceiling is the account's quota (requests and tokens per minute — watch EstimatedTPMQuotaUsage and InvocationThrottles), and the levers are cross-Region inference profiles to spread bursts, Provisioned Throughput to guarantee a floor, and the cost-side levers (caching, tiering) that shrink demand itself. On self-hosted endpoints (SageMaker AI), you own the fleet — and the scaling signal that works for LLMs is concurrency per model copy, not CPU.
| Serving mode | What scales | The signal | Idle behavior |
|---|---|---|---|
Bedrock on-demand | Nothing of yours — shared serverless capacity | Quota pressure: EstimatedTPMQuotaUsage, InvocationThrottles | Zero idle cost; the ceiling is the quota, not a fleet |
Bedrock Provisioned Throughput | Committed model units you purchase | Utilization of the committed units against the traffic floor | Billed whether used or not — right-size to the floor |
SageMaker real-time (inference components) | Model copies via the desired-copy-count scalable dimension | Concurrent requests per copy (high-resolution), not CPU | Minimum copies keep weights warm; cold starts cost minutes |
SageMaker asynchronous | Instances behind an internal queue | Queue backlog per instance | Scales to zero; queued requests wake it |
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.