Resilient request paths and model routing
Lesson 3 of 4 in FM APIs, Streaming, and Application Wiring.
On-demand FMs run on shared capacity behind quotas, which means throttling is a design input, not an incident. The kit, in order of engagement: the AWS SDK's exponential backoff (retries with exponentially growing, jittered waits — configurable retry modes, already built in) absorbs transient 429s; API Gateway rate limiting (throttling settings and usage plans with API keys) shapes your callers so one noisy client cannot spend everyone's quota; fallback mechanisms catch what retries cannot — another model, another Region (cross-Region patterns live in resilient model architectures), a cached or canned answer — so the product degrades gracefully instead of erroring; and AWS X-Ray traces the request across every service boundary, so when latency spikes you can see which hop — gateway, Lambda, model call, retry storm — actually spent the time.
One request survives a bad five minutes
- Request arrives
Through API Gateway, where usage plans already bounded this caller's rate — protecting the model quota from any single client.
- Call primary model
Converse/ConverseStream, traced by X-Ray so this hop's latency is attributable.
- Throttled or erroring?
- SDK backoff + jittered retries
Exponentially growing waits with jitter, capped attempts — built into the AWS SDKs. Absorbs transient throttles without hammering shared capacity.
- Recovered?
- Fallback path
A secondary model, a cross-Region profile, or a cached/simplified answer — graceful degradation that keeps the product alive at reduced fidelity.
- Answer returns
Primary or fallback — the caller gets a response either way, and X-Ray shows which path it took and what each hop cost.
Resilience decides whether a model answers; routing decides which one. Four rungs, in rising intelligence: static routing in application code or config — request type X goes to model Y, deterministic and auditable; Step Functions dynamic routing — a Choice state inspects content (language, size, classification) and branches to specialized FMs mid-workflow; Amazon Bedrock intelligent prompt routing — a managed router that predicts response quality per prompt and forwards each request to the model offering the best quality/cost combination in its family (default routers for the Nova and Meta families, or configured routers with your own quality-difference criteria and fallback model — and the response tells you which model served); and API Gateway request transformations, mapping and reshaping requests toward different backends at the edge of your API.
| Approach | Decision input | Where the logic lives | Exam cue |
|---|---|---|---|
Static configuration | Request type / tenant / feature flag | Application code or config | "Deterministic", "auditable mapping", "requirements rarely change" |
Step Functions content-based | The request's content, inspected mid-workflow | A Choice state branching to specialized FMs | "Route documents by language/type to specialized models", "within an orchestrated workflow" |
Predicted response quality vs cost, per prompt | Managed inside Bedrock — you call the router like a model | "Automatically pick the cheapest model that maintains quality", "no routing logic to build" | |
API Gateway transformations | Request shape at the API edge | Mapping/transformation rules in the gateway | "Transform and direct requests before they reach backends" |
Key terms: Exponential backoff, Intelligent prompt routing (Bedrock), AWS X-Ray
Interactive checkpoint quiz (3 questions) — open this page in a browser to take it.