Overfitting, underfitting, and what to do

Lesson 4 of 4 in Training and Tuning on SageMaker AI.

Fit problems are diagnosed from two curves, never one: training loss and validation loss. Overfitting (high variance): training keeps improving while validation stalls, then worsens — the model is memorizing noise. Underfitting (high bias): both curves sit high — the model never captured the signal in the first place. The gap between the curves is the diagnosis; the remedies for the two conditions are opposites, which is why the exam loves pairing them.

Line chart of loss versus epoch with two curves. The training loss curve falls steadily from about 2.2 toward 0.2 across ten epochs. The validation loss curve falls with it until roughly epoch 5, reaching about 0.9, then reverses and climbs back above 1.2 — the widening gap between the curves indicates overfitting.

The overfitting signature: training loss keeps falling while validation loss bottoms out around epoch 5 and climbs — the moment early stopping exists to catch. (illustrative — source: Amazon ML developer guide — model fit: underfitting vs. overfitting)

Diagnose from the curves, then prescribe
Condition	The curves show	Remedies
Overfitting (high variance)	Training loss low and falling; validation loss higher and rising	More/augmented data · regularization (L1/L2) · dropout · early stopping · simpler model · ensembling
Underfitting (high bias)	Both losses high and flat	Bigger/more expressive model · better features · train longer · raise learning rate · relax regularization

The anti-overfitting toolkit has a family name: regularization. L1 penalties push weights to exactly zero (built-in feature selection); L2 shrinks them smoothly; dropout randomly silences units during training so no single pathway becomes a crutch; early stopping simply quits at the validation minimum — the vertical line you would draw on the figure above. One more remedy deserves its own frame: combining models.

Bagging

Train many models in parallel on bootstrap samples; average or vote. Reduces variance — a committee of overfitters cancels out individual noise. Flagship: Random Forest. Ensembling cue: "reduce variance," "independent models in parallel."

Boosting

Train models sequentially, each one focusing on its predecessors' mistakes. Reduces bias — weak learners compound into a strong one. Flagship: XGBoost (yes, the built-in). Cue: "sequential," "each model corrects the last."

Stacking

Feed the predictions of diverse base models into a meta-model that learns how to blend them. Squeezes out the last points of accuracy at the cost of complexity and interpretability. Cue: "meta-model," "combine different model types."

Key terms: Overfitting, Underfitting, Regularization, Ensemble learning

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.