Inference economics: instances and purchasing options
Lesson 1 of 4 in Cost and Performance Engineering.
Training bills arrive in bursts; inference bills arrive forever — an endpoint meters every hour it exists, traffic or not, which is why inference usually dominates an ML system's lifetime cost. Domain 4 wants two reflexes: match the workload to the instance family, and match the commitment to the usage pattern. The exam never asks for dollar amounts — it asks which lever fits which scenario.
| Family | Built around | ML fit |
|---|---|---|
C (compute-optimized) | High CPU per GB of memory | CPU inference for small/classic models; feature pipelines |
R (memory-optimized) | Large memory per vCPU | Models or lookup tables that must sit in RAM; memory-hungry batch scoring |
G (GPU, inference-friendly) | NVIDIA GPUs balanced for serving | Deep-learning inference; small-scale training and fine-tuning |
P (GPU, training-class) | The biggest accelerators + fast interconnect | Large-scale distributed training — usually overkill for serving |
Inf (AWS Inferentia) | Purpose-built inference silicon (compile via Neuron SDK) | High-volume deep-learning inference at lower cost per inference |
Graviton (ARM, the "g" suffix) | ARM-based CPUs with strong price-performance | Cost-efficient CPU inference when your stack runs on ARM |
Then choose how you pay. On-Demand is the default: full flexibility, highest rate. SageMaker AI Savings Plans trade a 1- or 3-year committed spend for lower rates across SageMaker usage — the answer whenever a scenario says "steady, predictable usage for the next year". Spot uses spare capacity at a steep discount that can be interrupted — which is why it belongs to training (managed spot training checkpoints to S3 and resumes), and never to a real-time endpoint. For right-sizing, SageMaker AI Inference Recommender load-tests candidate instance types for endpoints, while AWS Compute Optimizer analyzes utilization and recommends right-sizing for EC2, Auto Scaling groups, EBS, and Lambda — the surrounding fleet of your ML system.
Pick the purchase option
Interactive decision tree — outcomes:
- Managed Spot Training
Checkpoint to S3, let SageMaker resume after interruptions, and take the discount. The exam tell: "can tolerate interruptions" or "flexible completion time".
- On-Demand training
A hard deadline makes interruption risk unacceptable. Right-size the instance and parallelize instead.
- SageMaker AI Savings Plans
Commit to steady usage, keep the same instances, pay less. Right-size before you commit — a Savings Plan locks in waste too.
- Serverless Inference
Scales to zero between bursts so idle hours cost nothing; accept cold starts. The tell: "intermittent", "unpredictable", "idle most of the day".
- On-Demand + auto scaling
Target-tracking auto scaling follows the curve (e.g. invocations per instance); provisioned capacity floors protect latency at peaks.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.