Evaluating Models — Classic and GenAI

Confusion matrices and the classic metric family, baselines and reproducible experiments with MLflow, GenAI evaluation from BLEU to LLM-as-a-judge, and shadow testing before launch.

Content current as of 2026-10.

Lessons

1. Confusion matrices and classic metrics
2. Baselines, experiments, and MLflow
3. Evaluating GenAI: BLEU, ROUGE, BERTScore, LLM-as-a-judge
4. Shadow variants and A/B before you ship

Sources

• SageMaker Autopilot — metrics and validation (metric definitions)
• Machine learning experiments with managed MLflow on SageMaker AI
• Amazon Bedrock — model evaluations
• SageMaker AI — shadow tests