The Serving Stack
Lesson 2 of 4 in Self-Hosting on EKS, AKS, GKE.
A self-hosted deployment is not “a model on a server” — it is five layers, each with its own failure modes and its own upgrade cadence. Bottom-up: GPU nodes, the driver-and-operator layer that makes those GPUs visible, the Serving engine that actually runs the model, the gateway that fronts it, and the observability that tells you whether any of it is working.
The foundation is the GPU node pool. All three managed Kubernetes services attach node pools built from their clouds’ accelerated instance families — on the menu verified for this domain (2026-09), that spans single-GPU L4- and A10-class inference nodes up to eight-GPU H100- and H200-class machines, plus AWS’s Trainium and Inferentia custom silicon with its own software stack. Do not memorize the menu; compare it in the GPU sizing tool when you need it.
Architecture stack diagram with five layers from bottom to top: GPU node pools (accelerated instances attached to EKS, AKS, or GKE), drivers and GPU operator (driver, device plugin, GPU monitoring), serving engine (vLLM-class engine with continuous batching, highlighted as the emphasis layer), API gateway and auth (authentication, rate limits, streaming), and observability (GPU, engine, and request metrics).
Two layers deserve a closer look because they carry the load. The driver-and-operator layer exists because a stock Kubernetes node knows nothing about GPUs: a documented operator mechanism installs the driver, runs the device plugin that advertises each GPU as a schedulable resource, and ships GPU telemetry. It is infrastructure software with its own version matrix — driver versions must stay compatible with both the node image below and the engine’s runtime above, which is why lesson 4 treats upgrades as day-2 work rather than an afterthought.
The serving engine is where the money is made or lost. It owns Continuous batching, KV cache paging, and multi-GPU parallelism — the mechanics that decide how many tokens of Throughput each billed GPU-hour yields. Everything about choosing and tuning that engine is taught once, in the inference domain’s batching-and-engines module; here the point is architectural: the engine is a pod like any other to Kubernetes, which is exactly what makes the whole approach portable. The weights it loads are your Open weights checkpoints or fine-tuned variants, pulled from your own artifact store — not from a provider’s Model catalog.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.