The Vendor Eval Table
Lesson 1 of 3 in In Production: Reading Model Evals Critically.
Every model launch comes with one: a table. Rows are Benchmark names — MMLU, HumanEval, a math suite, a long-context test. Columns are models: the new one, its predecessor, and a few competitors. Cells are scores, and the new model’s column is bold. You will read dozens of these tables in your career, and the whole Evaluation domain has been building toward reading them well.
The first thing to internalize: a benchmark score is a joint property of a model and a harness configuration. The Eval harness decides how many examples sit in the prompt (the shots), whether the model is prompted to reason step by step (Chain-of-thought (CoT)), the exact prompt template, the Sampling settings, how an answer gets extracted from raw output, and which version of the dataset is used. Change any of these and the number changes — same model, same benchmark name. That is why the table’s fine print is not decoration. Footnotes like 5-shot or 0-shot CoT are half of what the number means, and a version note can be the difference between an original test set and a cleaned or extended revision of it.
So read the table in two passes. Pass one, the headline pass everyone does: which column is bold, which rows does it win. Pass two, the engineer’s pass: for each element, ask the question that turns a claim into information.
| Table element | What it is claiming | The question to ask |
|---|---|---|
Benchmark name | This named test measures a capability you care about | What construct does this benchmark measure, and how close is that to my task? (The reading checklist from Benchmarks and Their Limits applies row by row.) |
The score itself | On this item set, under our harness configuration, the model achieved this value | Single run or averaged? What metric exactly — accuracy, pass@k at which k? Is the benchmark near saturation, where the last points are noise? |
Shot / CoT footnote | The prompt contained n examples; step-by-step reasoning was (or wasn’t) elicited | Do the footnotes match across every column — and do they match the other table I am about to compare this one against? |
Harness / version footnote | The number came from this evaluation setup or this dataset revision | Whose harness — the vendor’s own, or an independent one under standardized conditions? Which revision of the test set? |
Competitor columns | Rival models were measured for comparison | Measured by whom, when, under the same configuration — or copied from the rivals’ own cards with different footnotes? |
Now the second half of the anatomy: what the table does not contain. Three absences, all of them load-bearing.
No variance. The cell shows one number, not a range. Rerun the same eval with a reworded prompt template, a different shot selection, or nonzero Temperature and the score moves; the table does not tell you by how much. A one- or two-point gap between columns may be well inside that movement — treat close calls as ties unless someone shows you error bars.
No contamination status. You know from Data Contamination that web-scale training data can absorb test items, and that labs’ own papers document contamination analyses of their training corpora. A table cell does not tell you whether this model’s training data was audited against this benchmark, with what method, or what turned up. A score on a leaked test measures memory, not capability — and Contamination status is precisely the fine print that distinguishes them.
No “your task”. The most important absence. No vendor table has a row for summarizes our contracts without dropping clauses or answers our customers in our tone. The table samples the vendor’s choice of public capabilities; your workload is not in the sample frame. Nothing in the table is evidence about your task until your own Golden set makes it so.
Why one number hides a distribution
A benchmark score is an estimate built from many unstable ingredients, reported as if it were a constant. The instability comes from at least four places.
Prompt sensitivity. The template — how the question, choices, and instructions are laid out — is a free parameter, and models are not indifferent to it. Two reasonable templates can rank two models differently on the same items. The HELM effort (Liang et al., 2022) exists largely because of this: comparisons are only meaningful when scenarios, prompts, and metrics are pinned down and shared across all models being compared.
Sampling noise. If generation uses nonzero Temperature, the same model gives different answers across runs. Metrics that sample multiple completions, like pass@k, are defined statistically for exactly this reason — Chen et al. (2021) report an unbiased estimator for pass@k because the naive empirical estimate of “at least one of k passes” is biased when computed from a small number of samples. The definition of the metric, not just its value, is part of the claim.
Answer extraction. The model outputs text; the harness must decide what the model’s answer was. Scoring by comparing choice-token probabilities, by regex-matching a final answer, or by asking the model to emit a structured verdict are different measurements that disagree on real outputs — a model penalized for wrapping the right answer in prose loses points to parsing, not to ignorance.
Aggregation. Suite-style benchmarks average over subtasks. Whether that average weights each subtask equally or each item equally changes the headline number, and a model can win the average while losing the subtasks you care about. When a table cell summarizes dozens of subscores, ask which ones did the work.
None of this makes the numbers fake. It makes them point estimates with invisible error bars — useful for spotting large differences, unreliable for adjudicating small ones.
In production
Vendor eval tables follow models onto the cloud platforms: every model catalog surfaces cards with benchmark sections. The mechanism to internalize is provenance — the platform is displaying the vendor’s claims, not independently re-measuring them.
AWS
Amazon Bedrock’s model catalog spans many providers, and the benchmark numbers you encounter for those models originate from each provider’s own cards and papers — each measured under that provider’s harness choices. The catalog is a shortlisting surface: it tells you which models exist, their modalities, context lengths, and regions. Treat capability scores you meet there as vendor claims to be read with this lesson’s table anatomy, then measured against your own golden set through Bedrock’s own APIs before anything ships.
Azure
The Azure AI Foundry model catalog aggregates models from multiple publishers, and it also surfaces leaderboard-style benchmark views that let you sort and compare catalog models. The same two questions apply: whose harness produced each number, and are the configurations behind two models’ scores actually the same? Platform-surfaced comparisons compress the fine print by design — useful for narrowing a long list, never a substitute for the footnote check, and never evidence about your task.
Google Cloud
Vertex AI’s Model Garden presents first-party and open models side by side, each carrying documentation and evaluation claims produced by its publisher. Because one console shows many publishers, it is tempting to read across cards as if they were one table — resist exactly that. The uniform card layout does not mean uniform measurement conditions. Use the garden to shortlist on hard facts (modality, context window, region, license), then let your harness produce the numbers that decide.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.