Custom Silicon

Lesson 3 of 4 in GPUs and Accelerators.

Two of the three clouds sell accelerators they designed themselves, and both put them on the same instance menu you just learned to read. AWS lists Trainium for training — trn1.32xlarge carries 16 accelerators at 32 GB each, 512 GB per node — and Inferentia2 for inference, from a single-accelerator inf2.xlarge (32 GB) up to inf2.48xlarge with 12 accelerators (384 GB). Google sells the TPU, its long-running in-house accelerator line, consumed through Compute Engine, GKE, and Vertex AI; TPU families are documented separately from the GPU machine types, so treat their specs as a separate menu. As of October 2026 the newest generation is TPU7x (codename Ironwood, Google’s seventh), documented at 192 GB of HBM per chip — two chiplets of 96 GB — and ~7,380 GB/s per chip, in slices up to 9,216 chips, all on one machine type (tpu7x-standard-4t, four chips per VM). Two reading habits matter more than that number. First, generations coexist rather than replace each other: TPU7x, v6e, v5p, v5e and v4 are all still documented as current versions, so “the latest TPU” is a choice you make, not a default you inherit. Second, per-chip memory does not climb with the version number — v6e (Trillium) carries 32 GB per chip against the older v5p’s 95 GiB, because the e suffix marks the efficiency line and p the performance line. Read the suffix, not just the digit.

Azure is the instructive negative. Microsoft designs its own silicon and does ship some of it to customers: the Arm-based Cobalt 100 CPU has its own documented VM series (Dpsv6, Dplsv6, Epsv6 and relatives). Its AI accelerator, Maia, is a different story — as of October 2026 every accelerator in Azure’s GPU-accelerated families (NC, ND, NG, NV) is an NVIDIA or AMD part, and the VM size catalog documents no Maia-backed series at all. You cannot put one in a Terraform file. That gap is the lesson: an accelerator can be announced, fabricated, and running the vendor’s own services without ever becoming an instance type you can rent. The size catalog, not the keynote, decides what is on your menu this quarter — so check it before you plan a migration around a chip you have only read about.

The pitch for every one of these custom lines — Trainium, Inferentia, TPU — is price-performance: the vendor claims better throughput per dollar than comparable GPU instances for supported workloads. Read that sentence carefully — it is a vendor claim, measured on vendor-chosen workloads. Sometimes it holds for yours; the only way to know is to benchmark your model, your traffic shape, your precision. What the headline never prices is the second half of the trade: the software ecosystem is the real cost. The GPU world runs on CUDA — the default target of every Serving engine, kernel library, and research repo. Custom silicon runs on its own stack: AWS Neuron compiles models ahead of time for Trainium/Inferentia, and TPUs speak XLA through JAX and PyTorch/XLA. Whether your model architecture, your engine, and your quantization scheme are supported — today, at the version you need — is a per-model engineering question, not a checkbox.

So when does custom silicon earn its keep? The pattern is consistent: high, steady volume on a supported model architecture, inside a cloud you have already committed to. At that shape, the one-time engineering cost of the port amortizes across enough tokens to matter, the volume justifies real benchmarking instead of trusting claims, and the portability you gave up was not being used anyway. Inverting each condition tells you when to stay on GPUs: spiky or small workloads, exotic architectures, or a genuine multi-cloud posture. There is also a quieter, pragmatic reason teams try the custom-silicon queue: it is a different queue. Quotas are per instance family, so trn, inf, and TPU capacity is requested — and granted — separately from the GPU pool everyone else is waiting in. Lesson 4 makes that consideration systematic.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.