The Self-Hosted Cluster

Lesson 3 of 4 in Reference Architectures.

The third architecture replaces the managed model API with infrastructure you operate: Serving engine pods on GPU nodes, usually orchestrated by the managed Kubernetes service of your cloud (EKS, AKS, GKE — the mechanics live in Self-Hosting on Kubernetes). Everything around the model from lesson one survives unchanged — gateway, guardrails, output handling, evals. What changes is that the middle node explodes into a system whose throughput, latency, and unit cost are now your problem and your opportunity.

One quantity rules this architecture: Utilization. A managed API bills per Token; a cluster bills per GPU-hour whether tokens flow or not. Every box in this diagram exists either to produce tokens or to keep the GPUs that produce them busy.

The self-hosted cluster

  1. Client traffic

    Same contract as the managed-API app: the application talks to an endpoint and streams tokens back. Done well, application teams cannot tell the backend changed — that abstraction is what makes the migration in lesson four survivable.

  2. Load balancer

    The regional entry point: TLS termination and health-checked distribution across gateway replicas. Standard cloud plumbing — the LLM-specific logic lives one hop deeper.

  3. Inference gateway

    LLM-aware routing: which model, which tenant, which priority. It meters tokens per caller, sheds load when queues grow, and can route long-context requests to different pools than short chat turns. This is where you re-implement what the managed platforms bundled — service tiers, quotas, per-client limits.

  4. Serving-engine pods

    The token factory: an inference engine (vLLM is the canonical open example — Vertex AI’s own docs cite vLLM containers for self-deployed models) running Continuous batching, paged KV cache management, and optionally Quantization and Speculative decoding. One pod typically owns one model replica on one GPU or GPU group.

  5. GPU node pools

    Kubernetes node pools of GPU instances — sized with the GPU Sizing tool against the verified instance table. Platform mechanisms worth knowing qualitatively: the GPU device plugin/operator stack exposes GPUs to pods, and MIG or time-slicing can partition a large GPU for small models. Capacity itself is a resource: Capacity reservation for the baseline, Spot capacity only for interruption-tolerant work.

  6. Artifact registry + model store

    Container images in the registry; model weights (tens to hundreds of GB) in object storage. A new pod pulls both before serving its first token, so weight size sets your cold-start floor — keep weights in the same region as the nodes, or pay Egress and minutes.

  7. Observability

    Engine-level metrics, not just node-level: TTFT, TPOT, queue depth, KV-cache utilization, batch occupancy, and Goodput — throughput that met its latency target. Node CPU tells you almost nothing here.

  8. Autoscaling loop

    Consumes engine metrics and scales pods, then node pools. Documented platform mechanisms: horizontal pod autoscaling on custom metrics, plus cluster autoscaling for nodes. The catch is physics — a new GPU node must provision, pull images, and load weights, so scaling reacts in minutes. Scale ahead of predictable load; hold a warm buffer for the rest.

  9. Streamed response

    Tokens stream back through gateway and LB to the client. End-to-end, the request saw the same guardrail and output-handling layers as the managed-API app — self-hosting moved the model, not the responsibilities.

The diagram’s quiet dependency is the registry-to-engine edge. In the managed-API app, a model update is the provider’s problem; here it is a rollout: new weights land in object storage, pods restart in waves, and your Eval harness gates the wave — the same Regression testing discipline as a prompt change, now applied to multi-gigabyte artifacts. Teams that treat model files as “data” rather than versioned release artifacts discover the difference during their first bad rollback.

What changes at multi-region scale

The single-region diagram replicates; the interesting parts are what does not replicate for free.

Capacity stops being one number. GPU Quota is granted per region and per instance family, and physical availability differs by region — the GPU instance table is really a per-region availability question wearing a spec sheet. Multi-region means holding capacity (and often a Capacity reservation) in each region, sized for failover: two regions at 50% load each must each survive 100% when the other disappears.

Weights become a distribution problem. Every region needs a local copy of images and weights, or cold starts pay cross-region Egress and minutes. The registry edge from the diagram becomes a replication pipeline with its own lag — and a rollout is no longer “restart pods” but “drain a region, upgrade, re-run evals, shift traffic, repeat.”

Routing inherits the managed platforms’ homework. Geographic or latency-based routing at the global load balancer, session affinity so a conversation’s Prefix caching benefit survives, and an answer to the data-residency question — which is exactly the problem the managed platforms solve with Cross-region inference profiles, Data Zone deployments, and multi-region endpoints. Self-hosting means writing your own version of that policy. The failure patterns and runbooks get their own module: In Production: Multi-Region, Quota, and Failover.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.