Production Evals and Regression Testing

Evals as the gate: regression testing before any model or prompt change ships, quality monitoring after, and surviving vendor model updates.

Content last verified 2026-09.

Lessons

1. The Regression Gate
2. Monitoring Quality in the Wild
3. When Models Change Under You

Sources

• Liang et al. (2022) — Holistic Evaluation of Language Models (HELM): evaluating on many metrics at once
• Zheng et al. (2023) — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (judge biases)
• Chen et al. (2021) — Evaluating Large Language Models Trained on Code (sampling variance and pass@k)