Custom Silicon
Lesson 3 of 4 in GPUs and Accelerators.
Two of the three clouds sell accelerators they designed themselves, and both put them on the same instance menu you just learned to read. AWS lists Trainium for training — trn1.32xlarge carries 16 accelerators at 32 GB each, 512 GB per node — and Inferentia2 for inference, from a single-accelerator inf2.xlarge (32 GB) up to inf2.48xlarge with 12 accelerators (384 GB). Google sells the TPU, its long-running in-house accelerator line, consumed through Compute Engine, GKE, and Vertex AI; TPU families are documented separately from the GPU machine types, so treat their specs as a separate menu to verify. Azure’s menu, in our verified table, is all-NVIDIA.
The pitch for all of them is price-performance: the vendor claims better throughput per dollar than comparable GPU instances for supported workloads. Read that sentence carefully — it is a vendor claim, measured on vendor-chosen workloads. Sometimes it holds for yours; the only way to know is to benchmark your model, your traffic shape, your precision. What the headline never prices is the second half of the trade: the software ecosystem is the real cost. The GPU world runs on CUDA — the default target of every Serving engine, kernel library, and research repo. Custom silicon runs on its own stack: AWS Neuron compiles models ahead of time for Trainium/Inferentia, and TPUs speak XLA through JAX and PyTorch/XLA. Whether your model architecture, your engine, and your quantization scheme are supported — today, at the version you need — is a per-model engineering question, not a checkbox.
| Dimension | NVIDIA GPUs (all three clouds) | AWS Trainium / Inferentia | Google TPU |
|---|---|---|---|
Software ecosystem | CUDA — the default target of serving engines, kernels, and research code | Neuron SDK — ahead-of-time compilation; operator and model coverage must be verified per model | XLA via JAX and PyTorch/XLA — strongest where the stack was built with it in mind |
Serving-engine support | First-class in mainstream engines | Backend-specific — check your engine’s support for your exact model and features | Backend-specific — same verification burden, different stack |
Portability | The same stack runs on AWS, Azure, GCP, and on-prem | AWS only | Google Cloud only |
Procurement | The supply-constrained pool everyone is bidding on; quota per instance family | Separate instance families (trn, inf) with their own quotas — a different queue | Separate accelerator pool with its own quotas |
Economics | Market-priced scarcity; specs and prices fully public per instance | Vendor-claimed price-performance advantage — validate on your workload | Vendor-claimed price-performance advantage — validate on your workload |
So when does custom silicon earn its keep? The pattern is consistent: high, steady volume on a supported model architecture, inside a cloud you have already committed to. At that shape, the one-time engineering cost of the port amortizes across enough tokens to matter, the volume justifies real benchmarking instead of trusting claims, and the portability you gave up was not being used anyway. Inverting each condition tells you when to stay on GPUs: spiky or small workloads, exotic architectures, or a genuine multi-cloud posture. There is also a quieter, pragmatic reason teams try the custom-silicon queue: it is a different queue. Quotas are per instance family, so trn, inf, and TPU capacity is requested — and granted — separately from the GPU pool everyone else is waiting in. Lesson 4 makes that consideration systematic.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.