Why Kubernetes at All
Lesson 1 of 4 in Self-Hosting on EKS, AKS, GKE.
The previous module settled whether to self-host. This one is about how — and for most organizations that choose self-hosting, “how” lands on Kubernetes. Three forces push it there.
An existing platform team. If your organization already runs services on Kubernetes, a model server is, from the platform’s point of view, one more deployment: same CI/CD, same secrets management, same on-call rotation, same dashboards. The marginal cost of adding a GPU workload to a mature platform is far smaller than the cost of standing up a bespoke serving environment beside it.
Portability across clouds. The same manifests, Helm charts, and operators run on Amazon EKS, Azure AKS, and Google Kubernetes Engine — the three managed control planes, where the cloud operates the API server and etcd while you own the node pools and everything running on them. For teams with multi-cloud mandates, capacity constraints in one region, or negotiating leverage to protect, it matters that the serving stack is not welded to one vendor’s Managed endpoint shape.
Unified operations. One deploy pipeline, one observability stack, one incident process for the application and the model behind it. When the model is just another service in the mesh, nothing about running it is exotic to the rest of the organization.
| Dimension | Self-host on Kubernetes | Managed endpoint (dedicated) | Serverless API (pay-per-token) |
|---|---|---|---|
Control | Maximal — you pick the engine, its flags, the Quantization format, the upgrade moment, the placement | Model and capacity are yours; the serving internals are the provider’s | Model choice and request parameters only — everything else is opaque |
Ops burden | Maximal — drivers, engine, gateway, observability, capacity, and incidents are all yours | Provider runs the serving; you manage provisioned capacity and quotas | Minimal — you manage quotas, retries, and a bill |
Portability | High — manifests and engine move across EKS, AKS, GKE, and on-prem clusters | Low — the endpoint shape, units, and APIs are provider-specific | Low-to-medium — APIs differ, though gateways can abstract the request shape |
Capacity responsibility | Yours end to end: node pools, reservations, headroom, scale-down discipline | Shared: you buy units or reserved capacity; the provider delivers them | The provider’s — you inherit their quotas and their throttling behavior |
Where it wins | Existing platform team, Open weights or custom models, strict control or portability requirements, high sustained load | Steady traffic on catalog models with predictable throughput needs | Spiky or low traffic, fastest time to first request, smallest team |
Read the matrix column by column and the trade is plain: Kubernetes buys the most control and the most portability at the price of the most operations. Nothing in this module makes that trade cheaper — it makes it concrete, layer by layer, so you can price it before you sign up for it.
Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.