Capacity Reality

Lesson 4 of 4 in GPUs and Accelerators.

The spec sheet tells you what exists. Quota decides what you get. All three clouds gate accelerator instances behind per-region, per-family quota systems with documented increase-request processes — and for the larger GPU families, default allocations are commonly too low to launch anything, so the request is not paperwork, it is a project dependency. Approval takes review time, is not guaranteed, and arrives family by family and region by region.

Quota is also only permission. In a supply-constrained market, approved quota does not promise that physical capacity exists in your zone at the moment you scale out — insufficient-capacity errors at launch time are a documented failure mode on every cloud. That gap between permission and supply is exactly what the procurement modes below exist to manage.

You can hold accelerator capacity three ways, and the difference is who carries the risk. On-demand is pay-as-you-go with no promise that instances will be available when you ask — you carry the supply risk, per scale-out event. Reservations (Capacity reservation mechanisms, offered in various term and calendar-window forms) shift supply risk to the provider: the capacity is yours for the covered period — and so is the bill, whether you use it or not, which makes Utilization the number that decides whether a reservation was a bargain or a write-off. Spot (Spot capacity) inverts the deal: steep discounts because the provider can reclaim the instance on short notice. Spot is a fit exactly where interruption is cheap — checkpointed training and fine-tuning, batch inference, or stateless serving replicas fronted by a load balancer with an on-demand or managed-endpoint fallback — and a trap everywhere else.

Choosing a capacity strategy

Interactive decision tree — outcomes:

  • Reserve the baseline, overflow to on-demand

    Cover the traffic floor with reserved capacity and let peaks spill to on-demand (or a managed per-token endpoint). Watch utilization: a reservation only beats on-demand when it is actually busy — that math is the Cost Modeling module’s job.

  • On-demand with a tested fallback

    Autoscale on-demand, and rehearse the failure you will eventually see: an insufficient-capacity error at scale-out. The fallback — second zone or region, second instance family, or managed per-token endpoint — must be config, not a redesign.

  • Design to the quota you can get

    File the increase request now, then re-size honestly: quantize to fit a smaller or more available family, split across regions or families, or launch on a managed per-token endpoint while the request is pending. Capacity you cannot get is not architecture — it is a wish.

  • Spot-first, checkpointed, with on-demand fallback

    Take the discount: interruptions cost only the work since the last checkpoint. Keep an on-demand path for deadline pressure, and make resume automatic — a spot strategy that needs a human to restart jobs is a pager strategy.

  • Reserve a window for the run

    Non-checkpointable work needs guaranteed capacity for its whole duration — this is what calendar-style reservation mechanisms are for. Better still: make the job checkpointable and unlock the spot economics next time.

In production

Capacity planning is part of architecture now. The teams that ship design for the quota they can get — not the instance they want — file increase requests at design review, and treat a capacity denial as a first-class failure mode with a rehearsed fallback.

AWS

Accelerated instances are gated by per-region service quotas on each family (p, g, trn, inf), with increases requested through Service Quotas and subject to review. The production pattern: request quota the week the design is drafted, keep a second family fully sized as fallback (a g6e fleet for when p5 capacity is not there), and treat an insufficient-capacity error in one Availability Zone as a routing event — the autoscaling group should already span zones and know its alternate instance types. The custom-silicon families are a separate quota queue, which is sometimes reason enough to qualify a model on inf2.

Azure

GPU VM series draw on per-region vCPU quota per VM family, requested through the portal, and access to the largest ND-series sizes can require additional justification. Teams that survive contact with this design multi-region from day one — the same template deployed to a second region turns a slow or partial quota grant into a deployment parameter instead of a redesign — and they use the fractional-GPU NVadsA10 sizes to keep dev and test environments from consuming the production family’s quota.

Google Cloud

Accelerator quotas are per region and per accelerator type, managed through the console’s quota pages, and the largest deployments are typically arranged through reservation mechanisms rather than raw on-demand. The pragmatic pattern: size against the g2 and a2 tiers you can actually launch today, keep the a3-class request in flight, and let a managed per-token endpoint absorb what the fleet cannot serve yet — the fallback doubles as the migration path if self-hosting stops paying.

Key terms: Quota, Capacity reservation, Spot capacity, Utilization

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.