Confusion matrices and classic metrics

Lesson 1 of 4 in Evaluating Models — Classic and GenAI.

Every classification metric on the exam is four numbers wearing different outfits. The confusion matrix counts predicted-vs-actual: TP (caught it), FP (false alarm), FN (missed it), TN (correctly ignored it). Work one example until it is reflex. A fraud model scores 1,000 test transactions, of which 50 are actually fraud. It flags 70 as fraud: TP = 40, FP = 30, and it misses FN = 10, leaving TN = 920.

A two-by-two confusion-matrix heatmap. Rows are the actual classes (fraud, legitimate); columns are the predicted classes (fraud, legitimate). Cell values as shares of all 1,000 transactions: actual fraud predicted fraud 0.04 (40 true positives); actual fraud predicted legitimate 0.01 (10 false negatives); actual legitimate predicted fraud 0.03 (30 false positives); actual legitimate predicted legitimate 0.92 (920 true negatives). The true-negative cell visually dominates, illustrating class imbalance.

The worked confusion matrix as shares of 1,000 test transactions: TP 40 (0.04), FN 10 (0.01), FP 30 (0.03), TN 920 (0.92). Derived metrics: accuracy 0.96, precision 40/70 ≈ 0.57, recall 40/50 = 0.80, F1 = 2·(0.57·0.80)/(0.57+0.80) ≈ 0.67. The huge TN cell is why accuracy flatters: a model that never flags anything would still score 0.95. (calculated — source: SageMaker Autopilot — metrics and validation (metric definitions))

The metric family — what each answers, and when each one lies (worked values from the figure above)
Metric	Formula & worked value	It answers	It lies when
Accuracy	(TP+TN)/all = 960/1000 = 0.96	How often is the model right overall?	Classes are imbalanced — "always legit" scores 0.95 here
Precision	TP/(TP+FP) = 40/70 ≈ 0.57	When it flags fraud, how often is it right?	Used alone: ignore recall and you can be precise by barely ever flagging
Recall	TP/(TP+FN) = 40/50 = 0.80	Of all real fraud, how much did it catch?	Used alone: flag everything and recall hits 1.0
F1	2·P·R/(P+R) ≈ 0.67	Are precision AND recall both decent?	Costs of FP and FN are wildly asymmetric — then weight them explicitly
AUC-ROC	Area under the TPR-vs-FPR curve across all thresholds	How well does the model rank positives above negatives, threshold-free?	You need performance at one operating threshold, or classes are extremely skewed
RMSE / MAE	Root-mean-squared / mean-absolute error	Regression: how far off are predictions, in the target's units?	Mixed up with classification — these apply to numeric targets only; RMSE punishes outliers harder

Interactive sorting exercise: Pick the metric each scenario should optimize or report.

Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.