Apples to Apples Is Hard
Lesson 2 of 3 in In Production: Reading Model Evals Critically.
Lesson one taught you to read one table. The trouble starts when you put two tables side by side — Vendor A’s card against Vendor B’s card — and do what every spreadsheet instinct screams to do: subtract.
The subtraction assumes the two numbers came from the same measurement. Across vendors, they almost never did. Each vendor runs its own Eval harness, with its own shot counts, its own prompt templates, its own Chain-of-thought (CoT) policy, its own answer extraction, its own choice of dataset revision and subsets, its own aggregation — and each vendor, entirely rationally, publishes the configuration under which its model looks best. Neither number is false. The comparison is.
This is the mechanism behind a pattern you will see constantly: model launches where each of several vendors appears to lead the field on the same benchmark family. No one needs to cheat for this to happen — selecting favorable shots, templates, and metric variants per vendor is enough. The standardized-evaluation projects exist precisely as the antidote: HELM (Liang et al., 2022) made the point by re-evaluating many models under one shared harness — same scenarios, same prompts, same metrics — so that a difference between two rows could finally be attributed to the models rather than to the measurement.
Cross-vendor tables are one instrument. You actually have three, and they measure different things.
Benchmark tables score fixed items against answer keys under one harness configuration. Strengths: reproducible, cheap to check, capability-specific. Weaknesses: everything this domain has taught — saturation, gaming, Contamination, configuration sensitivity.
Arena leaderboards aggregate human preference votes on live, uncontrolled prompts. Platforms like Chatbot Arena (Chiang et al., 2024) collect pairwise battles — two anonymous models answer the same user prompt, the user picks a winner — and fit a rating from the win/loss record, in the Elo rating family descended from chess ratings (Bradley–Terry-style models of pairwise outcomes). Strengths: real prompts, real preferences, hard to game by memorizing a test set. Weaknesses: it measures what voters prefer on their prompts — a construct entangled with style, formatting, and confidence, and one that need not transfer to your workload. And when LLM-as-judge stands in for the crowd, the judge’s own biases — position, verbosity, self-preference — come with it (Zheng et al., 2023); calibrating judges properly is a discipline of its own, covered in depth by our sister AI Agent Academy.
Your evals score your Golden set with your Rubric under the configuration you ship. Weaknesses: small, yours to build and maintain. Strength: it is the only instrument pointed at the construct your decision is actually about.
When the three disagree — and they will — the disagreement is data. An arena darling that stumbles on your golden set is telling you your task is not what the crowd was voting on.
| Instrument | What it measures | What moves the number | Right use |
|---|---|---|---|
Benchmark table | Accuracy on fixed, answer-keyed items under one harness configuration | Model capability — but also shots, templates, extraction, contamination, dataset revision | Tracking capability trends; large-gap screening; never adjudicating small gaps across harnesses |
Arena leaderboard | Aggregated human preference over live, uncontrolled prompts (Elo-family rating from pairwise votes) | Answer quality — but also style, verbosity, formatting, refusal behavior, and the prompt mix voters bring | Sensing broad assistant quality and momentum; shortlisting generalists |
Your eval harness | Your task, on your traffic, with your rubric, at your shipped configuration | The thing you actually care about — plus the quality of your golden set and graders | The decision itself: model selection, regression gating, vendor-update audits |
Interactive sorting exercise: Sort each claim: is it a fair apples-to-apples comparison as stated, or not comparable without more information?
In production
Comparison shopping is a first-class activity on the clouds, and each platform has surfaces that make cross-model comparison look easier than the measurement fine print warrants. The mechanism that protects you is always the same: compare through one harness — yours — over the access path you will ship.
AWS
On Amazon Bedrock, the practical apples-to-apples move is structural: because many providers’ models sit behind one converse-style API, you can run your identical golden set, prompts, and parameters across candidates and diff the outputs — your own standardized conditions. Bedrock’s model evaluation jobs support exactly this pattern, running candidate models over a shared prompt dataset with shared metrics, which is a far stronger comparison than reading two providers’ cards against each other.
Azure
Azure AI Foundry’s catalog includes benchmark-comparison views that rank models side by side. Read the methodology notes before trusting a ranking: whatever harness produced those numbers is a fixed public configuration, not your workload. The catalog comparison shortlists; the Foundry evaluation tooling — running your dataset with your evaluators against each shortlisted deployment — is where a defensible comparison happens, because both models finally pass through one measurement.
Google Cloud
Vertex AI’s evaluation services are built around exactly the mechanism this lesson teaches: side-by-side comparison, where two models (or two prompts, or two configurations) answer the same inputs and a judge or metric scores each pair under identical conditions. That design encodes the lesson’s rule — never compare numbers born in different harnesses; put both candidates through one harness and compare outputs, with your golden set as the input.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.