Hyperparameters and automatic model tuning
Lesson 2 of 4 in Training and Tuning on SageMaker AI.
Parameters are what training learns; hyperparameters are what you set before it starts. Three of them do most of the exam's work. Learning rate — the step size of each update: too high and loss oscillates or diverges, too low and training crawls or stalls in a bad spot. Batch size — examples per update: small batches are noisy but generalize well; large batches are stable and fast on big hardware but can land in sharper minima and need learning-rate adjustments. Epochs — full passes over the data: too few underfits, too many overfits.
| Knob | Raise it → | Lower it → | Classic failure |
|---|---|---|---|
Learning rate | Faster progress, risk of overshooting | Stable but slow; may stall | Loss explodes or see-saws → rate too high |
Batch size | Smoother gradients, better hardware use, more memory | Noisier updates, sometimes better generalization | Out-of-memory errors → batch too large for the GPU |
Epochs | More chances to learn — and to memorize | Cheaper, but may stop short of convergence | Validation loss rising while training loss falls → too many |
Hand-tuning these is guesswork, so SageMaker AI ships Automatic Model Tuning: declare ranges for each hyperparameter and one objective metric, and AMT launches a fleet of training jobs to search. Strategies: Bayesian (the default — treats search as regression, each completed trial informs the next pick), random, grid, and Hyperband (starts many trials, aggressively early-stops the losers — strong for deep nets where bad configs reveal themselves early). Warm start seeds a new tuning job with knowledge from previous ones; early stopping kills trials whose objective is lagging the best so far.
One AMT iteration (Bayesian strategy)
- Define ranges + objective metric
- AMT picks a candidate config
Bayesian: a surrogate model predicts which untried configuration is most promising given all completed trials.
- Run a training job
A normal training job — channels, container, CloudWatch — with the candidate hyperparameters injected.
- Objective metric lands
Read from the job's emitted metrics; lagging trials can be early-stopped.
- Trial budget left?
- Best training job wins
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.