Day 2: The Work That Never Ends

Lesson 4 of 4 in Self-Hosting on EKS, AKS, GKE.

Getting the stack up is day 1. Day 2 is everything after, and it is where self-hosting budgets fail — because the recurring work is real but rarely written into the plan. It comes in four streams.

Upgrades on three clocks. Serving engines ship fast and fixes land in new releases, so staying current is not optional. But the engine pins a runtime version, the runtime constrains the driver, and the driver ties to the node image — a version matrix, not a version. Driver and node-image upgrades mean draining GPU nodes, and draining a node that holds an expensive, slow-to-refill accelerator is a capacity event, not a routine reboot.

Model artifact management. Weights are multi-gigabyte binaries that must be versioned like code: an artifact registry, immutable version identifiers, and checksums verified at deploy time, so “what exactly is serving right now?” always has an answer and rollback always has a target. The file formats those artifacts ship in — and what Quantization variant you serve — are covered in the inference domain’s quantization module; the pipeline below is where those choices get enforced.

A model-rollout pipeline on Kubernetes

  1. New model or engine version

    A new fine-tune, a new base checkpoint, a new quantization variant, or a serving-engine upgrade — all take the same path.

  2. Push immutable artifact

    Weights land in the artifact store under an immutable version id with a checksum; engine images are pinned by digest, never by a mutable tag like latest.

  3. Deploy to staging pool

    Same manifests, smaller GPU pool. Deploy-time checksum verification catches corrupt or wrong-version artifacts before they touch traffic.

  4. Eval gate passes?

    An eval-harness regression suite compares quality against the current version, plus latency and throughput checks — engine upgrades can change outputs even with identical weights.

  5. Canary on a traffic slice

    A small share of real traffic hits the new version while dashboards compare TTFT, TPOT, error rates, and quality signals against the baseline.

  6. Canary healthy?

    Regression in latency, cost per token, or output quality stops the rollout here — with a small blast radius.

  7. Full rollout

    The new version becomes the pinned production artifact; the old one stays in the registry as the rollback target.

  8. Roll back to pinned version

    Possible in one step only because versions are immutable and the previous artifact is still addressable by checksum.

Incident response, with new failure modes. LLM serving adds pages classic runbooks do not cover: out-of-memory on long-context requests, KV cache pressure collapsing batch sizes (throughput drops while GPUs look busy), a single hot node degrading Goodput, and engine regressions that only Regression testing against a pinned Eval harness suite will catch before users do.

Cost visibility — where Utilization becomes THE metric. A self-hosted GPU has no per-token meter; the meter is the clock. Your effective cost per token is a formula with named variables: cost_per_token = (gpu_hours_billed × price_per_gpu_hour) ÷ tokens_served. The numerator runs 24/7 for every node you hold; only the denominator responds to traffic. Idle GPUs are therefore not a rounding error — they are the largest single lever on the unit economics, which is why per-team GPU accounting, scale-down discipline, and an idle-GPU dashboard belong in the first sprint, not the retrospective. The break-even arithmetic against managed options is worked through in the cost-modeling module; the sizing side lives in the GPU sizing tool.

In production

All three managed Kubernetes services document GPU node pools and the operator pattern for node-level GPU software — the day-2 mechanics in this lesson are shared across them. What differs is the instance menu behind the node pool (compare it in /tools/gpu-sizing) and the local flavor of the tooling.

AWS

Amazon EKS documents GPU-accelerated node groups on EC2 accelerated instances, with GPU-enabled node images and the device-plugin/operator pattern for drivers and telemetry. The menu behind the pool is EC2’s accelerated families — including Trainium and Inferentia silicon, which swap the CUDA stack for the Neuron SDK, changing the engine layer, not the Kubernetes mechanics.

Azure

Azure AKS documents GPU node pools on the N-series VM sizes, with a documented path for installing and managing the GPU driver stack. Azure’s menu is distinctive at the small end — documented fractional A10 VM sizes sell a partial GPU as the instance — while the drain-and-upgrade discipline for driver and node-image updates is the same as everywhere else.

Google Cloud

Google Kubernetes Engine documents GPU node pools on its accelerated machine families with driver installation options managed by the platform, and its managed-mode offerings still surface the same realities: whole-device scheduling, slow node provisioning, and hourly billing while a GPU is held. The mechanics transfer; only the menu and the defaults change.

Key terms: Managed endpoint, Serving engine, Continuous batching, Utilization, Capacity reservation

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.