Auto scaling and instance choices

Lesson 4 of 4 in Serving Predictions: Endpoints and Batch.

Endpoints scale through Application Auto Scaling: you register the variant's instance count as a scalable target with a min and max, then attach a policy. The policy the docs recommend — and the exam rewards — is target tracking on SageMakerVariantInvocationsPerInstance, the average number of invocations per minute each instance receives. Set the target from a load test ("one ml.c6i handles 70 requests/min at acceptable latency → target 70") and SageMaker AI adds instances as traffic climbs and removes them as it falls, respecting your cooldowns.

Line chart of invocations per minute over 24 hours with two series. A smooth request-load curve rises from about 40 overnight to a midday peak near 480, then falls back through the evening. A stepped capacity line, equal to instance count times the 70-invocations-per-minute target, trails it: flat at 70 overnight, stepping up through 140, 210, 350, 420, and 490 during the morning climb, with each step arriving slightly after the load has crossed the previous capacity level; holding 490 through the midday peak; then stepping down through 420, 350, 280, 210, and 140 across the afternoon and evening, visibly later than the load falls, and returning to 70 — one instance — at night.

A day of target tracking on SageMakerVariantInvocationsPerInstance with a 70-invocations/min target. The stepped line is serving capacity — instance count × 70 — so each step of 70 is one instance. Steps up lag the climbing load (new instances take minutes to provision, pull the image, and load the model), the fleet holds through the peak, and scale-in steps down one instance at a time, later and more cautiously, under the scale-in cooldown. The floor is one instance — reactive scaling never reaches zero. (illustrative — source: SageMaker AI developer guide — automatic scaling of SageMaker AI models)

Target tracking — the default answer

Pick a metric (SageMakerVariantInvocationsPerInstance, or CPU/GPU utilization as a custom metric) and a target value; Application Auto Scaling does the arithmetic in both directions. Scale-in and scale-out cooldowns stop it thrashing on noisy traffic.

Step scaling — custom thresholds

Explicit CloudWatch alarm thresholds mapped to capacity steps ("latency > 500 ms → add 2 instances"). More control, more to maintain — the stem usually has to ask for threshold-level control before this beats target tracking.

Scheduled scaling — known spikes

Raise the minimum capacity ahead of a predictable event (market open, campaign launch) and lower it after. Reactive policies only act after load arrives; scheduled scaling is how you beat a spike you can see coming.

What auto scaling cannot do

Scale a real-time endpoint to zero — the floor is one instance per variant. Scale-to-zero economics belong to serverless and asynchronous inference.

Instance families for inference — conceptual, no price sheets on the exam
Family	Silicon	The fit
ml.c / ml.m / ml.r	CPU	Classical ML (XGBoost, scikit-learn, linear models) and light traffic — most tabular models never need more
ml.g / ml.p	NVIDIA GPU	Deep learning at latency or batch sizes a CPU cannot hold; g-family for inference economics, p-family for the heaviest models
ml.inf (Inferentia)	AWS Inferentia (purpose-built inference chip)	High-throughput deep learning inference at lower cost than GPUs — models are compiled for it with the AWS Neuron SDK

Key terms: Real-time inference, Serverless inference, Asynchronous inference, Batch transform, Production variant, Multi-model endpoint (MME), Inference pipeline

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.