The Shortlist Protocol
Lesson 2 of 3 in In Production: Reading a Model Card and Choosing a Model.
The instinct is to ask “which model is best?” The protocol asks a different question first: what must be true? Five requirement axes decide most choices before any benchmark is consulted:
- Modality — what must go in (text, images, audio, video) and what must come out. A Multimodal requirement instantly eliminates most of the landscape.
- Context — how much input the workload actually needs and how much output, measured on real traffic, not guessed. Cards state these separately for a reason.
- License constraints — what your legal posture tolerates: permissive only? Community license terms acceptable? Attribution obligations shippable?
- Deployment shape — API-only, self-hosted on your infrastructure, or Edge deployment on constrained hardware. Open weights vs Closed weights is decided here, not by ideology.
- Cost ceiling — a number per request or per month, derived from unit economics. Ceilings turn “cheaper is better” into a pass/fail filter.
Write these down as a one-pager before looking at any model. Requirements written afterward have a way of drifting toward whatever the shiniest candidate happens to offer — the one-pager exists to be un-driftable.
The shortlist protocol, end to end
- Write the requirements one-pager
Modality in/out, context in/out, license constraints, deployment shape, latency and cost ceilings. Mark each requirement hard or soft. Do this before looking at models.
- Filter the family map
Model cards, license pages, and the Cloud Model Matrix — eliminate on hard constraints only. Modality and license filters are fast and objective.
- More than 3 survivors?
A long shortlist means the requirements are not doing their job yet.
- Tighten a soft requirement into a hard one
Promote the constraint you believe most — usually cost ceiling or deployment shape — and re-filter.
- Shortlist 2–3 candidates
Include the incumbent or platform default if one exists, plus the strongest challenger. Record why each survived.
- Run YOUR eval harness on each
Same golden set, same graders, same configuration for every candidate — the machinery from Building an Eval Harness.
- Does a candidate clear your bar?
The bar was set in the one-pager: quality threshold on the golden set, within the cost and latency ceilings.
- Revisit the one-pager
No candidate clearing the bar is information: a requirement is too tight, the task needs decomposition, or the landscape is not there yet. Change the requirement explicitly — never silently.
- Pick with evidence; file the runner-up
The decision record: one-pager + filter results + harness numbers. The runner-up is your pre-evaluated fallback when the winner changes or retires.
The license filter deserves its own drill, because it is where fast reading goes wrong. “Open-weights” names a distribution fact, not a legal one — the license decides what you may build, and licenses vary within families and within catalogs.
The permissive end is genuinely simple: the gpt-oss-120b card states a “Permissive Apache 2.0 license”, and MIT plays the same role elsewhere. The custom end carries obligations you must ship with: the Llama 4 Community License Agreement requires “Built with Llama” attribution, derived model names starting with “Llama”, and a separate license for entities exceeding 700 million monthly active users (license page) — and the weights are gated behind accepting “the license terms and acceptable use policy” (HF org). The Gemma Terms of Use include a Prohibited Use Policy and reserve Google’s right to “remotely restrict violating usage” — while the same terms page notes that Gemma 4 has a separate Apache 2 license (terms). One family, two license regimes, depending on generation.
Mixed catalogs are the rule, not the exception. Mistral’s official models overview tags open models variously Apache 2.0, “Modified MIT” (with “exceptions for companies with large revenue”), and CC BY-NC 4.0 — three legal postures inside one organization (docs). And license metadata is verified per model card: Qwen’s flagship card lists apache-2.0 and DeepSeek’s lists MIT, but each of those was read on one specific card — the org pages name no licenses at all (Qwen, DeepSeek). That is exactly why the protocol’s rule is mechanical: read the license of the exact artifact you will ship. Family reputation licenses nothing.
| Requirement | Where you verify it | What verification looks like (documented examples) |
|---|---|---|
Modality | The card’s input/output modality statement — read both directions | Llama 3.2 Vision’s card table: input “Text + Image”, output “Text”. Qwen2.5-Omni-7B’s card: perceives “text, images, audio, and video” while “generating text and natural speech responses in a streaming manner”. Same word — multimodal — very different envelopes. |
Context | The card’s context lines — noting native vs extrapolated, input vs output | gemma-3-4b-it documents input (128K for 4B/12B/27B, 32K for 1B) and output (8192) separately; SmolLM3-3B states “Trained on 64k context and supports up to 128k tokens using YARN extrapolation”; Qwen3-0.6B states a single “Context Length: 32,768”. Three conventions — normalize before comparing. |
License constraints | The license page or card metadata for the exact model version | Apache 2.0 / MIT at the permissive end (gpt-oss; per-card Qwen and DeepSeek entries); community terms at the other — attribution and naming rules, a 700M-MAU threshold (Llama 4), use policies and remote-restriction rights (Gemma Terms of Use). |
Deployment shape | Weights availability plus your platform’s catalog — the Cloud Model Matrix crosses both | Open-weights families are downloadable (sometimes gated behind accepting terms); the GPT, Claude, and Gemini API families are closed-weights — consumable only as a service, per their vendors’ docs pages. |
Cost ceiling | Not on the card — measured from your traffic against platform pricing, e.g. via the Token Cost Estimator | Cards state sizes and active-parameter counts (gpt-oss-120b: “117B parameters with 5.1B active parameters”) — inputs to a cost model, never the cost itself. Your tokens-in/tokens-out distribution decides. |
Writing the model-requirements one-pager
One page, six sections, written before any model is named. A workable template:
- Task and quality bar. One sentence per task the model must perform, and the Golden set metric that defines “good enough” — the number the harness will later be held to. If you cannot state the bar, you are not ready to shortlist.
- Modality envelope. Inputs and outputs, listed separately. Include the ugly ones now (scanned PDFs are images; voice notes are audio) — modality retrofits are re-platforming events.
- Context arithmetic. Percentile input sizes from real or simulated traffic, plus required output length. Compare against the same quantities on candidate cards — native vs extrapolated, input vs output.
- License constraints. What legal has actually approved: permissive only, or specific community terms reviewed and accepted? Name the obligations you can ship (attribution? naming rules?) and the ones you cannot.
- Deployment shape and data boundary. Where prompts and outputs may travel — API acceptable, or weights inside your VPC, or on-device? This one line usually halves the landscape.
- Ceilings and tie-breakers. Cost per request, latency budget — as numbers. Then the soft preferences (Reasoning model effort dials, Fine-tuning support, ecosystem maturity) that break ties among candidates that clear every hard bar.
Mark every line hard or soft. Hard lines filter; soft lines rank. The discipline of the one-pager is that changing a line after seeing candidates must be done in writing, in front of the same people who approved it — drift dies in daylight.
In production
Every managed platform ships machinery for exactly this protocol: catalogs built for filtering, and evaluation tooling for running your harness against a shortlist.
AWS
On Amazon Bedrock the filter step is the model catalog (by provider, modality, region) plus per-model access grants; the harness step maps to Bedrock’s model evaluation capability, which runs automatic and human-review evaluations — including with your own prompt datasets — across candidate models. The protocol’s decision record fits naturally in the artifacts: access request, evaluation job results, and the chosen model ID pinned in configuration.
Azure
In Azure AI Foundry the catalog’s filters (collection, capability, deployment type) implement the requirements filter, and Foundry’s evaluation tooling scores shortlisted models against your own data and graders. Because catalog entries carry per-provider terms, the license line of your one-pager gets checked per entry — and the deployment-shape line maps to the entry’s available deployment options, serverless or managed.
Google Cloud
On Google Cloud, Model Garden is the filterable map — first-party, open, and partner models with card-style entries — and Vertex AI’s generative-AI evaluation service runs side-by-side comparisons of candidates on your prompts with your criteria. Open models deployed to managed endpoints keep upstream license obligations, so the one-pager’s license line survives deployment unchanged.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.