Selecting foundation models: benchmarks, capabilities, tradeoffs
Lesson 2 of 4 in Designing GenAI Solutions and Selecting Models.
Model selection is examined through three different instruments: performance benchmarks, capability analysis, and limitation evaluation. Benchmarks (public leaderboards, task-specific suites) are a cheap first filter across the catalog. Capability analysis checks the non-negotiables: modality (text, vision, embeddings), context window, tool/function-calling support, languages, customization options. Limitation evaluation is the professional half everyone skips: maximum output length, latency profile, Regional availability, quota ceilings, data-handling terms.
The verdict, though, never comes from a leaderboard: it comes from your evaluation on your data. Bedrock model evaluation runs candidate models against your task set with automatic metrics, human review workflows, or LLM-as-a-judge scoring — turning the PoC's anecdotes into a comparable score per model.
| Axis | The question it answers | Where to check |
|---|---|---|
Capability fit | Can it do the task at all — reasoning depth, vision, tool use, languages? | Model card capabilities + a task-sample PoC |
Context window | Do the documents, history, and retrieved chunks fit per request? | Model card limits vs your real payload sizes |
Latency profile | Does P95 meet the SLO at production payload sizes? | Measured in the PoC — never assumed from model size |
Token economics | What does a request cost at projected volume? | Input/output token counts from the PoC × published rates |
Customization support | Does it support fine-tuning / adapters if the roadmap needs them? | Bedrock customization support per model; SageMaker for full control |
Hard constraints | Regional availability, residency, quotas, provider terms | Regional model listings + service quotas — filters, not preferences |
Interactive sorting exercise: Sort each requirement into what it is for model selection: a hard constraint that filters the catalog, a capability the model must demonstrate, or an economics question your PoC must measure.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.