Quotas as Architecture
Lesson 1 of 3 in In Production: Multi-Region, Quota, and Failover.
Every architecture in the previous module was drawn as if capacity were infinite. It is not — and on managed platforms the finiteness has a precise, documented shape: the Quota. All three clouds meter model inference in tokens and requests per unit time, scoped to a region and a model, and all three throttle you when the meter trips. Amazon Bedrock enforces quotas on token usage per model — tracked separately for each of its two inference endpoints — with a documented process for requesting increases. Azure AI Foundry assigns quota per subscription, per region, per model, per deployment type, in tokens-per-minute that you allocate to deployments; exceed a deployment’s allocation and requests receive HTTP 429. Vertex AI enforces per-project, per-region, per-base-model request metrics through the Cloud Quotas system, distinguishing adjustable quotas from fixed system limits.
The teams that suffer treat these numbers as an ops afterthought — something to raise via support ticket after the first outage. The teams that do not suffer treat them the way structural engineers treat load ratings: a property of the material the design must work within. A quota is not an obstacle to your architecture. It is one of its inputs.
| Amazon Bedrock | Azure AI Foundry | Vertex AI | |
|---|---|---|---|
Unit of the meter | Quotas on token usage for model inference, viewable in the Service Quotas console and the Bedrock service quotas pages | Tokens-per-minute (TPM) allocated to each deployment, which also sets a proportional requests-per-minute (RPM) limit — the RPM:TPM ratio varies by model | Per-metric request quotas such as |
Scope | Per model, tracked separately for each of the two inference endpoints ( | Per subscription, region, model, and deployment type — you distribute the allocation across deployments and can redistribute it between deployments of the same model | Per project, region, and base model — a tuned model shares its base model’s quota rather than getting its own |
At the limit | Throttling, with a documented quota-increase request process | HTTP 429; a dynamic-quota preview can let standard deployments burst above configured TPM when spare platform capacity exists | Resource-exhausted (429) errors; the global endpoint is documented as reducing them, and batch draws on a separately allocated shared pool with no predefined quota limits |
Buying certainty | Provisioned Throughput purchased in Model Units, each defining per-minute input- and output-token processing | Provisioned deployment types purchased as PTUs that guarantee a processing level; batch has its own enqueued-token quota so it cannot starve online traffic | Provisioned Throughput purchased as GSUs for a commitment term; overage above the reserved baseline bills as pay-as-you-go and can be controlled per request |
Three design responses turn a quota from a wall into a parameter.
Request-shaping spends the meter deliberately. Every token you do not send is quota you do not consume: trimmed context, capped max_tokens, cached prefixes (Prefix caching discounts exist because the meter is tokens), and per-tenant budgets that stop one customer from exhausting everyone’s allocation. Measure your real traffic shape with the Token Cost Estimator — the same numbers that drive cost drive quota.
Queuing reshapes when tokens are spent. Per-minute meters punish spikes, not volume, so an admission queue that smooths a bursty arrival curve into a steady flow can serve the same traffic without a single 429. Bound the wait and protect Goodput: work that can tolerate minutes does not belong in the interactive path at all — all three platforms document cheaper lanes for it (batch tiers everywhere; Bedrock and Vertex additionally document Flex tiers for latency-tolerant traffic and Priority tiers for the opposite end).
Spreading multiplies the meters. Because quota is scoped per region — and on Azure per deployment, on Bedrock per endpoint — running the same model in two places doubles the ceiling. The classic split is a reserved baseline sized to steady-state demand with spillover to on-demand for peaks: Vertex documents exactly this pattern (Provisioned Throughput baseline, overage to pay-as-you-go), and it is the standard shape on the other two as well. Utilization decides whether the reserved slice pays for itself — that arithmetic lives in Cost Modeling.
A request through quota-aware routing
- Incoming request
Arrives tagged with a priority class (interactive, background, batch) and an estimated token budget — the two facts the router needs before touching any meter.
- Shape the request
Trim retrieved context, cap
max_tokens, reuse cached prefixes. Every token not sent is quota not spent — the cheapest capacity there is. - Admission: headroom now?
Compare live token/request meters against the deployment’s allocation. Interactive traffic proceeds; background work waits; batch-class work exits to the platform’s batch tier entirely.
- Queue — smooth the burst
A short, bounded queue converts a spiky arrival curve into a flow the per-minute meter accepts. Past the bound, degrade rather than pile up — an unbounded queue converts an overload into a latency outage.
- Primary: reserved baseline
A Provisioned throughput baseline sized to steady-state demand serves most traffic at fixed cost and predictable latency.
- Throttled (429)?
The platform’s signal that this meter, here, is spent. The signal is expected at peaks — what matters is having a next hop wired.
- Spill to on-demand
The documented pattern on all three platforms: reserved capacity covers the baseline, pay-per-token absorbs the peak. Vertex additionally lets you control per request whether overage may spill.
- Still throttled?
On-demand quota is a regional meter too. When both meters in one region are hot, the remaining headroom is geographic.
- Cross-region route
The platform mechanisms of the next section: Bedrock inference profiles, Azure Global / Data Zone deployments, Vertex multi-region or global endpoints — other regions’ capacity, inside your residency boundary.
- Response served
Log which path served it. The distribution across these routes is your early-warning dashboard: rising spillover share is tomorrow’s capacity decision.
- Degradation ladder
Smaller model, cached answer, graceful refusal — designed in lesson 2, not improvised mid-incident.
Spreading needs a mechanism, and this is where the platforms have converged on the same idea with three vocabularies: let one logical endpoint draw on more than one region’s capacity. Each version of Cross-region inference trades along the same axis — routing freedom buys capacity and availability; routing control preserves data-residency guarantees. The tabs below give the mechanism as each platform documents it.
Amazon Bedrock
Cross-Region inference works through inference profiles, in two documented forms. Geographic profiles (US, EU, APAC) keep routing within a geography — the documented answer to data-residency needs. Global profiles let Bedrock route a request to any supported commercial AWS Region, documented at approximately 10% savings versus standard pricing.
The operational fine print, per the User Guide: routing itself incurs no additional cost, destination Regions do not need to be manually enabled in your account, traffic stays on the AWS network encrypted in transit, and the Region that actually processed each request is logged in CloudTrail. One structural caveat shapes capacity planning: inference profiles do not currently support Provisioned Throughput — reserved capacity stays a per-Region asset.
Azure AI Foundry
The routing dial is the deployment type, chosen when you deploy a model. Global types dynamically route traffic to available datacenters in any Azure region. Data Zone types confine processing to a Microsoft-defined US, EU, or Asia Pacific data zone. Standard and Regional Provisioned types keep processing within a customer-specified Azure geography.
Two documented facts to anchor on: data stored at rest always remains in the designated geography — the deployment type varies where processing happens, not storage — and new models roll out Global first, then Data Zone, then geography-based types, with no guaranteed availability date for the regional tail. Tighter residency therefore also means later model access.
Vertex AI
Three documented serving surfaces: regional endpoints, us/eu multi-region endpoints that keep ML processing of customer data within a jurisdictional boundary such as the United States or European Union, and a global endpoint that “covers the entire world,” improves availability, and reduces resource-exhausted (429) errors.
The docs warn against the global endpoint when you have ML-processing residency requirements, because you cannot control which region processes a request. It also carries documented feature limitations — no tuning, no batch prediction for Anthropic/open MaaS models, no RAG corpus — and supports Provisioned Throughput only for an enumerated set of Gemini models. Check the current list against your workload before adopting it as the failover surface.
In production
Quota is the first production constraint a managed-model workload hits — before cost, usually before latency. What the meter is and where the headroom lives, per platform:
AWS
Bedrock quotas meter token usage per model and are tracked separately per inference endpoint (bedrock-runtime and bedrock-mantle), so instrument both if you use both. Defaults vary with account factors; increases go through the documented Service Quotas process — file them from measured traffic before launch, not after the first throttle. Beyond on-demand limits, headroom is Provisioned Throughput in Model Units (a per-Region purchase) or a cross-Region inference profile that spreads requests across a geography.
Azure
TPM is a budget you allocate: the subscription’s per-region, per-model, per-deployment-type quota gets divided across your deployments, and each deployment’s share is its rate limit (with a proportional RPM ceiling — the ratio varies by model). Redistribution between deployments of the same model is a routine, documented operation — rebalancing TPM toward the hot deployment is often the fastest headroom there is. Watch 429 rates per deployment, and keep batch work on batch types, which draw on a separate enqueued-token quota.
Google Cloud
Quota is enforced per project, per region, per base model — and a tuned model shares its base model’s quota, so a fine-tuned rollout does not bring fresh headroom. Adjustable quotas are raised through the Cloud Quotas console with an approval loop; fixed system limits are not negotiable, so check which kind you are near. For guaranteed capacity, the documented pattern is a Provisioned Throughput (GSU) baseline with overage spilling to pay-as-you-go, controllable per request.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.