How to Check a Capability Claim

Lesson 4 of 4 in Capabilities and Hard Limits.

Capability claims arrive daily: a vendor deck says the model “passes professional exams”, a paper reports a record Benchmark score, a teammate saw a demo that “basically already does our workflow”. After three lessons of mechanism, you can now read these claims the way you would read any measurement — by interrogating how it was made. Four questions do most of the work.

Who measured it? A maker reporting on its own model has chosen the tasks, the prompts, and the presentation. That does not make the number false — it makes it a best case, tuned by parties with every incentive and every ability to tune. Independent measurements with published methodology deserve more weight; screenshots of one great completion deserve none, because with Sampling in the loop, one impressive output is one draw from a distribution you have not seen.

On what data? A score is a property of a specific test set, metric, and prompt setup. “92% on a coding benchmark” tells you about that benchmark’s task distribution — short, self-contained problems, mostly in popular languages. Whether any of it transfers to your inputs is precisely what the jagged frontier says you cannot assume.

Could the test have leaked into training? Public benchmarks live on the same internet that training corpora are scraped from — questions and answers. A model can score well partly by having effectively seen the test. This failure mode is called contamination; one line is all it gets here, because the Evaluation domain owns the full treatment. The practitioner’s reflex: the more famous and public the benchmark, the more skeptically you treat a high score.

Does it match your distribution? Your traffic has its own languages, formats, edge cases, and failure costs. A claim measured on someone else’s distribution is evidence about theirs. The only measurement that settles your adoption decision is one made on inputs sampled from your own workload.

Walk a capability claim to a verdict

Interactive decision tree — outcomes:

  • Trust — then verify small

    Independent measurement, plausible data hygiene, matching distribution: this is as good as secondhand evidence gets. It still earns only a pilot — run a small evaluation on your own traffic before you commit an architecture to it.

  • Treat as a best case — test first

    A maker’s own number is an upper bound produced under conditions chosen by the maker. Reproduce the measurement on a sample of your real inputs; expect the gap between their number and yours to be informative.

  • Adjacent is not equal — test first

    The jagged frontier’s core rule: success on a neighboring task licenses no conclusion about yours. The claim justifies curiosity and a pilot, never a rollout.

  • The score may be memory — test first

    If the test could have been in training data, part of the score may be recall of the answers rather than ability — contamination. Re-measure on data the model cannot have seen: yours. The Evaluation domain covers detection in depth.

  • Interesting, not evidence

    A solid result on a task you will never run is trivia for your decision. File it away; do not let it move an adoption choice.

  • Ignore — until someone shows the setup

    A number without task, data, metric, and prompt setup is marketing. Ask for the setup; if it never arrives, the claim was not a measurement.

Most claims land on “test first”, so know what that costs: less than you think. A few dozen real examples from your own traffic, each with a known-good answer, beats any leaderboard for your decision — assemble them once, run every candidate model against them, and keep the set. It becomes your regression suite for the frontier: when a new model version ships, you re-run it and learn within the hour whether your tasks moved inside or outside. Building bigger, defensible evaluations — metrics, judges, statistical care — is the Evaluation domain’s whole curriculum; the habit starts here.

In production

The jagged frontier is an operational fact, not a philosophical one: model behavior differs across tasks and shifts across versions. Mature platforms therefore treat a model like any other dependency — it must pass your tests before it ships, and something must catch it when it fails anyway.

AWS

Amazon Bedrock provides model evaluation jobs that run candidate models against prompt datasets — including ones you supply — and score them before anything reaches users. Because many models sit behind one API, teams stage a model change like a code release: run the evaluation suite against the incumbent and the candidate, promote on evidence, and keep a fallback model wired for when a task regresses.

Azure

Azure AI Foundry builds evaluation into the deployment workflow: evaluators score model outputs on your own data for quality and safety, and those runs slot into CI/CD so a model swap is gated the way a code change is. Versioned deployments make the rollback path a configuration change rather than a rewrite — the safety net jaggedness demands.

Google Cloud

Vertex AI offers a generative-AI evaluation service that measures models against criteria you define on datasets you bring — the institutional form of “test on your own distribution”. Endpoint-based serving supports splitting traffic between model versions, so a new model earns its rollout gradually while the incumbent keeps handling the rest.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.