Reading a Benchmark Number

Lesson 4 of 4 in Benchmarks and Their Limits.

Everything so far compresses into a checklist you can run in under a minute whenever a score crosses your desk. Five questions, in order:

1. Which harness, which settings? Is the evaluation code named and versioned? Were all compared models run through the same harness with the same settings — or is this a table quilted together from different reports?

2. How many shots, what format, CoT or not? Shot count, prompt format, and whether Chain-of-thought (CoT) was allowed each move scores on their own. A number without these footnotes is not yet a measurement — it is a headline.

3. Which benchmark version and subset? Benchmarks get revised, filtered, and subsetted; “the same benchmark” across two reports may be two different task sets. Version identifiers matter exactly the way dependency versions do.

4. What is the contamination status? Was the test set checked against the training data? At web scale, overlap is the default expectation, not a rare accident — the data contamination module covers how leakage happens and how it is detected.

5. Is the delta bigger than the spread? Sampled decoding, nondeterministic serving stacks, and harness updates all make repeated runs differ. Few published tables report variance, so apply the working rule: small deltas between strong models are ties until someone shows otherwise.

Interrogating a claimed score

  1. A claimed score lands on your desk

    A launch post, a leaderboard screenshot, a vendor table — a number, a benchmark name, and a comparison.

  2. Same harness and settings for every model compared?

    Named, versioned eval code; identical settings across models. Tables assembled from different reports fail here immediately.

  3. Shots, prompt format, and CoT disclosed?

    Each of these moves scores by itself. Undisclosed conditions make the number unreproducible — and unfalsifiable.

  4. Benchmark version and subset named?

    Revisions, filtered subsets, and translated variants all travel under the original name.

  5. Contamination addressed?

    Any overlap analysis between the test set and the training data? Silence at web scale means “probably some”.

  6. Delta larger than run-to-run spread?

    If no variance is reported, judge conservatively: treat small gaps between strong models as ties.

  7. Treat as evidence — then confirm on your own evals

    A clean chain of disclosures earns provisional trust. The migration decision still belongs to evals on your tasks, your data, your harness.

  8. Treat as marketing until rerun in a harness you control

    Not an accusation — just the correct prior for an unreproducible number produced under selection pressure.

One more limit, even for a number that passes every check: it is still one benchmark. The foundations domain’s capabilities-and-limits module showed that model ability is jagged — strong and weak in patterns that do not follow human intuition — so a model that wins on MMLU can lose on your extraction task, and vice versa. A benchmark score is a sample from that jagged surface at one point; your product lives at a different point. Scores from preference arenas and LLM-as-judge pipelines get their own module later in this domain, with their own bias checklist; and where those judge scores concern agent trajectories — multi-step tool-using runs — evaluation and judge calibration for them is the sister AI Agent Academy’s territory, not this site’s.

In production

Every cloud model catalog shows you benchmark tables. They are built from exactly the choices this module covered — harness, shots, version, selection — usually by the model’s own vendor. The evergreen mechanism on every platform is the same: use catalog numbers to shortlist, then run your own evaluation on your own tasks before committing.

AWS

Amazon Bedrock’s catalog surfaces provider-reported benchmark figures on model detail pages — numbers produced by each provider’s own harness and settings, so a cross-vendor table is rarely settings-matched. Bedrock’s model evaluation tooling exists for the step that actually decides: running candidate models over the same prompt set, same settings, your data, inside your account.

Azure

Azure AI Foundry’s model catalog includes benchmark comparison views that aggregate published scores across models. Treat them as directional shortlisting aids — the harness and shot conditions behind entries differ — and use Foundry’s evaluation tooling to score shortlisted models on your own dataset under one configuration before switching a workload.

Google Cloud

Vertex AI Model Garden spans first- and third-party models whose cards cite each publisher’s own benchmark results — same caveat, different logo. Vertex AI’s evaluation services support side-by-side comparison of models on your prompts with consistent settings, which is precisely the settings-matched rerun this lesson keeps demanding.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.