Scheduling GPUs

Lesson 3 of 4 in Self-Hosting on EKS, AKS, GKE.

To the Kubernetes scheduler, CPU is a fluid: divisible into millicores, overcommittable, throttleable. A GPU is a brick. The device plugin advertises GPUs as countable devices; pods request them as integers; and the documented behavior is blunt — no fractions, no oversubscription, no sharing between containers by default. A pod that asks for one GPU takes the whole thing, idle or not.

Bricks make packing hard. Picture a pool of eight-GPU nodes: a pod requesting four GPUs, another requesting three, and a third requesting two cannot all land on one node — someone waits, or a GPU strands. Multiply by real fleets and fragmentation becomes a standing tax: the cluster reports free GPUs, yet a pending pod cannot use them because they are scattered one-here-two-there across nodes. Scheduling niceties like bin-packing policies and topology awareness reduce the waste; they do not repeal it.

Heatmap with three rows labeled Node A, Node B, and Node C, and eight columns labeled GPU 1 through GPU 8. Node A shows all eight cells fully allocated. Node B shows six cells allocated and two cells empty, illustrating stranded capacity. Node C shows four cells allocated, two cells at half intensity representing time-sliced sharing, and two cells empty.

Toy bin-packing picture: three eight-GPU nodes after a day of scheduling. Node A is fully packed; node B strands two GPUs that no pending multi-GPU pod can use; node C time-slices two GPUs across small workloads. The numbers are invented for teaching. (illustrative — source: Kubernetes documentation — Schedule GPUs)

When whole GPUs are too big a unit, two documented sharing mechanisms split the brick — with very different guarantees. MIG (Multi-Instance GPU) partitions a supported GPU in hardware into slices with their own compute and memory; each slice appears to Kubernetes as a separate schedulable device, and neighbors are isolated from each other. Time-slicing shares one GPU in software by interleaving workloads on it — cheap and universal, but with no memory isolation: one greedy neighbor can starve or crash another. The menu itself sometimes does the splitting for you — Azure’s documented fractional A10 VM sizes sell a partial GPU as the instance.

For LLM serving, sharing mechanisms matter less than you might expect: a well-run Serving engine is already a multiplexer, packing many requests onto one GPU through Continuous batching. MIG and time-slicing earn their keep on the edges — dev and staging environments, small auxiliary models like embedding models or rerankers, and bursty internal tools that would otherwise strand a whole device.

Why GPU autoscaling lags CPU autoscaling

CPU autoscaling feels instant because the expensive step — provisioning a node — is usually amortized away: a horizontal pod autoscaler places new replicas on warm nodes in seconds. GPU serving rarely has that luxury; the replicas are the nodes, and adding one is a pipeline of slow, sequential steps:

  1. Find capacity. Accelerated instances are supply-constrained; without a Capacity reservation, the provision request can be rejected or queued, and Spot capacity can be reclaimed mid-flight.
  2. Boot and join. The node boots, joins the cluster, and gets its GPU software — driver, device plugin, telemetry — installed or validated by the operator.
  3. Pull images. Serving-engine containers bundle CUDA runtimes and are multi-gigabyte; a cold pull takes minutes.
  4. Load the model. Tens to hundreds of gigabytes of weights stream from object storage into VRAM, then the engine warms up — allocating KV cache pools and compiling kernels.

Every step is minutes-shaped, so the pipeline is minutes even on a good day — which has two design consequences. First, scale on leading indicators, not lagging ones: queue depth and request arrival rate move before GPU Utilization does, and TTFT breaches mean you were already late. Second, shorten the pipeline you cannot skip: pre-baked node images with drivers installed, pre-pulled engine images, weights cached on fast local disk or a warm volume, and scheduled scale-up ahead of predictable peaks. Teams that treat GPU autoscaling like CPU autoscaling discover the difference as an SLO breach, every morning, at the same time.

Interactive checkpoint quiz (1 questions) — open this page in a browser to take it.