Multi-model and multi-container endpoints
Lesson 3 of 4 in Serving Predictions: Endpoints and Batch.
One model per endpoint stops scaling the moment you have one model per tenant: 500 fine-tuned XGBoost models would mean 500 idle fleets. SageMaker AI has three consolidation patterns, and the exam tests whether you can tell them apart by two facts: how many containers are involved, and who picks the model on each request.
A multi-model endpoint (MME) runs one container that serves many model artifacts from an S3 prefix. Models load lazily on first invocation and SageMaker AI evicts least-recently-used ones when memory fills — the caller names the artifact per request with TargetModel. A multi-container endpoint (MCE) runs up to 15 different containers — different frameworks side by side — invoked individually via TargetContainerHostname. An inference pipeline chains containers in sequence on the same endpoint: output of one is input to the next, so preprocess → predict → postprocess becomes a single InvokeEndpoint call (and the same chain works in batch transform).
| Pattern | Containers | Who picks what runs | Reach for it when |
|---|---|---|---|
Multi-model endpoint | One shared serving container | Caller, via | Many models, same framework, each too lightly used to deserve a fleet |
Multi-container (direct) | Up to 15 different containers | Caller, via | A few models on different frameworks sharing one instance |
Inference pipeline | Containers in a fixed sequence | Nobody — every request runs the whole chain | Preprocessing/postprocessing must travel with the model (e.g., a scikit-learn featurizer feeding XGBoost) |
Separate endpoints | One per model | The application, by URL | Each model needs its own scaling profile, hardware, or isolation |
Interactive sorting exercise: File each serving scenario under the pattern it is describing.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.