The Menu
Lesson 1 of 4 in GPUs and Accelerators.
When you self-host, the instance catalog is where architecture meets procurement. All three clouds publish that catalog in the same grammar: an instance name, an accelerator model, how many of them, memory per accelerator, and memory for the whole box. Learn to read one provider’s menu and you can read them all — AWS under its accelerated-computing instance families (p, g, trn, inf), Azure under its GPU-accelerated VM sizes (ND, NC, NV series), Google Cloud under accelerator-optimized machine families (a2, a3, a4, g2).
The menu is long, but it collapses into four tiers. Frontier nodes — H100, H200, and B200-class parts, usually eight to a box with NVLink-class interconnect — exist for training and for serving the largest Open weights models with Tensor parallelism. Mainstream instances — A100 and L40S-class — carry most self-hosted inference and Fine-tuning. Entry instances — L4 and A10-class, down to Azure’s fractional-GPU sizes that expose one-sixth of a card — serve small or heavily quantized models, embedding workloads, and low-traffic endpoints. Custom silicon — AWS Trainium and Inferentia, Google TPU — is a separate menu with its own trade, and gets lesson 3 to itself.
| Tier | AWS | Azure | Google Cloud | Typical LLM use |
|---|---|---|---|---|
Frontier | p5 / p5e / p5en — 8× H100 (80 GB) or H200 (141 GB) | ND H100 v5 / ND H200 v5 — 8× SXM with NVLink; NC H100 v5 — 1–2× H100 NVL (94 GB, PCIe) | a3 — 1–8× H100/H200 SXM; a4 — 8× B200 (180 GB); a4x — GB200/GB300 superchips | Training; tensor-parallel serving of the largest open-weights models |
Mainstream | p4d / p4de — 8× A100 (40/80 GB); g6e — 1–8× L40S (44 GiB per AWS docs) | NC A100 v4 — 1–4× A100 80GB PCIe; ND(m) A100 v4 — 8× A100 with NVLink | a2 — 1–16× A100 (40/80 GB) | Most self-hosted inference; fine-tuning mid-size models |
Entry | g6 — L4 (22 GiB per AWS docs); g5 — A10G (22 GiB per AWS docs) | NVadsA10 v5 — A10 (24 GB), including 1/6, 1/3, and 1/2 fractional GPUs | g2 — 1–8× L4 (24 GB) | Small or quantized models, embeddings, dev and low-QPS endpoints |
Custom silicon | trn1 / trn1n — 16× Trainium (32 GB); inf2 — 1–12× Inferentia2 (32 GB) | No custom-silicon rows in our verified table | TPU families — documented separately from the GPU machine types | Cost-motivated scale where the software stack supports it (lesson 3) |
Reading a row correctly matters more than memorizing the rows. Two habits keep you honest. First, the memory figure you can use is the one the provider documents, not the chip’s nominal capacity: AWS lists usable GiB that differs from nominal specs — the L40S shown as 44 GiB, the A10G and L4 as 22 GiB — while Google Cloud documents its L4 at 24 GB. Second, totals are the provider’s own figures, not always count × per-GPU arithmetic: AWS documents g6e.48xlarge at 357 GB total across 8 L40S, and g6.48xlarge at 178 GB across 8 L4 — neither is what naive multiplication gives you. Size against the documented row; the calculator below does.
Horizontal bar chart of total accelerator memory in GB for ten AWS instances: p5en.48xlarge with 8 H200 at 1128, p5.48xlarge with 8 H100 at 640, p4de.24xlarge with 8 A100 at 640, trn1.32xlarge with 16 Trainium at 512, inf2.48xlarge with 12 Inferentia2 at 384, g6e.48xlarge with 8 L40S at 357, g6.48xlarge with 8 L4 at 178, g6e.xlarge with 1 L40S at 44, inf2.xlarge with 1 Inferentia2 at 32, and g6.xlarge with 1 L4 at 22. The chart shows the roughly fifty-fold spread of the menu on a single cloud.
Tool: GPU Sizing Calculator — Pick a model, a precision, and a context budget — the GPU Sizing Calculator computes the memory footprint and maps it onto every verified instance row on AWS, Azure, and GCP.
Interactive checkpoint quiz (2 questions) — open this page in a browser to take it.