Baselines, experiments, and MLflow
Lesson 2 of 4 in Evaluating Models — Classic and GenAI.
A metric means nothing without a baseline: the majority-class guesser, the naive "tomorrow = today" forecast, the current rule-based system. Establish that floor first — a model that cannot beat it has no business in production, however sophisticated its training. Then make every comparison reproducible. Managed MLflow on SageMaker AI is the tool for that, and the one to recognise in a stem: experiments group runs, each run logs its parameters, metrics, and artifacts, the UI compares runs side by side, and the winner registers into the model registry.
| Piece | What it records |
|---|---|
Experiment | A named group of related runs — "churn-model-v2 candidates" |
Run | One training execution: its params, metrics over time, and artifacts |
Params | The inputs: hyperparameters, data version, feature set |
Metrics | The outputs: loss curves, validation scores — comparable across runs |
Artifacts | Files a run produced: the model, plots, evaluation reports |
Model registry | Versioned, promotable record of the models worth keeping |
The experiment loop
- Baseline run
Majority-class guess, naive forecast, or the current heuristic — trained (or computed) first and logged like any other run. This is the floor.
- MLflow tracking
Every run joins the experiment with its params (hyperparameters, data version, feature set) and metrics — recorded, not remembered.
- Candidate runs
New features, algorithms, hyperparameters — change one thing per run so comparisons stay causal.
- Compare runs
Side by side in the MLflow UI: does any candidate beat the baseline on the metric that matters?
- Register the best
The winner is versioned into the Model Registry, promotable toward deployment — and becomes the new floor the next loop must beat.
Experiments also catch training that never converged. Three loss-curve pathologies, three suspect lists. Flat from the start: learning rate too low, broken labels or features, a model too small for the task. Oscillating or exploding: learning rate too high, bad batches, numeric instability. Train falls while validation rises: overfitting — handled in Training and Tuning. There is a fourth curve that looks like a triumph:
Loss hits ~zero almost immediately — celebrate?
Investigate. Suspiciously instant convergence usually means data leakage: the target (or a proxy of it) is hiding in the features, or test rows leaked into training. The model aces evaluation and faceplants in production.
Loss never drops at all
Check the boring things first: labels aligned with features? learning rate orders of magnitude too small? data actually shuffled? Then consider capacity — a linear model cannot learn a nonlinear boundary no matter how long it trains.
Loss oscillates wildly
Lower the learning rate first; it explains most oscillation. Then look at batch size (tiny batches = noisy gradients) and outliers or corrupt rows arriving in some batches.
Validation metric varies hugely between reruns
Small validation sets and unseeded randomness make model comparison a coin flip. Fix seeds where supported, use enough validation data, and compare runs in MLflow rather than from memory.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.