Model Deployment and Throughput Strategies

Where the model runs and how much it can take: Bedrock on-demand versus provisioned throughput versus SageMaker endpoints, LLM-specific container sizing, model cascading, and task-specific small models.

Content current as of 2026-10.

Lessons

1. Three ways to host a model
2. Sizing for LLMs: memory, GPU, token capacity
3. Cascading and task-specific small models

Sources

• AIP-C01 exam guide — Content Domain 2 (tasks 2.1–2.5)
• Amazon Bedrock user guide — Provisioned Throughput
• Amazon SageMaker AI — model parallelism and large model inference
• Amazon Bedrock user guide — intelligent prompt routing