Splits, shuffles, and the leakage trap

Lesson 3 of 5 in Data Quality, Labeling, and Bias.

Three sets, three jobs. Training fits the parameters. Validation tunes what training cannot — hyperparameters, early stopping, model selection. Test gets read once, at the end, as the honest estimate; consult it repeatedly and it quietly becomes a second validation set. Before splitting: shuffle (datasets arrive ordered by time, class, or region, and an unshuffled split misrepresents all three), stratify when classes are imbalanced (preserve the ratio in every set), and for time series split by time — train on the past, test on the future, never randomly.

Then comes the trap the exam loves: leakage — information that will not exist at prediction time finding its way into training.

The taxonomy, with the concrete tells. Target leakage — a feature that encodes the outcome because it was recorded after it: account_closed_date in a churn model, amount_refunded in a fraud model. Temporal leakage — randomly splitting time-stamped data, so training rows postdate test rows. Preprocessing leakage — fitting scalers, encoders, or imputers on the full dataset before splitting. Duplicate leakage — the same (or nearly the same) record on both sides of the boundary, including synthetic copies made by oversampling before the split.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.