Most teams treat running LLMs on Kubernetes as a scaled-up version of running a web service. Package the model in a container, request a GPU, add a Horizontal Pod Autoscaler, and call it done. That approach survives a demo. It usually starts to break down under sustained production traffic for the same three reasons: GPU scheduling, model loading time, and autoscaling logic that was never designed for workloads this heavy.
This is not a "Kubernetes is great for AI" article. Kubernetes is a reasonable place to run generative AI workloads, but only if you plan around how differently they behave from the stateless services the platform was originally built for.
Why generic Kubernetes patterns fall apart for LLM workloads
A typical Kubernetes pod starts in seconds, uses a predictable and small amount of memory, and can be killed and rescheduled with little consequence. An LLM inference pod behaves nothing like that. It can take minutes to start because it needs to locate or download model weights and load them into GPU memory. Whether those weights come from an image layer, object storage, or a shared volume, startup time is usually much longer than a typical stateless service. It needs a specific, expensive, and often scarce piece of hardware. Killing it mid-request does not just drop a connection. It wastes a request that was already consuming GPU cycles for tens of seconds, sometimes longer for multi-step agent workloads.
None of this is exotic knowledge to anyone who has run inference workloads at scale. It is, however, the part most "how to deploy AI on Kubernetes" content skips, because it is easier to write about YAML manifests than about scheduling contention and GPU memory fragmentation.
GPU scheduling is not pod scheduling
The default Kubernetes scheduler treats a GPU as a single indivisible resource requested through the NVIDIA device plugin. A pod either gets a whole GPU or it gets nothing. For a large training job that is fine. For inference, it is usually wasteful, because most inference workloads do not saturate a modern GPU's compute or memory.
Two mechanisms fix this, and each has a real cost.
Time-slicing lets multiple pods share a GPU by dividing compute time between them. It is simple to configure and works with almost any workload, provided the combined workloads fit comfortably within the GPU's available memory. But it gives no memory isolation. One noisy pod can starve another's memory allocation, and a pod that leaks GPU memory will eventually take the others down with it too.
Multi-Instance GPU (MIG), available on newer NVIDIA data center GPUs, partitions a physical GPU into isolated instances with dedicated memory and compute slices. This is safer for multi-tenant clusters. The partition sizes are fixed at boot, though, and cannot be resized without draining the node. Sizing a MIG profile wrong means either wasted capacity or an outage when a model does not fit the slice it was assigned.
There is no universal right answer here. Teams running many small models for different internal customers usually want MIG for isolation. Teams running one or two large models at high throughput often get more value from dedicated GPUs or model-level batching, because GPU memory capacity usually becomes the limiting factor before compute utilization does.
Multi-tenancy and GPU isolation for untrusted workloads
Everything in the previous section assumes some baseline trust between the workloads sharing a GPU. That assumption breaks the moment a cluster serves multiple external customers, or internal teams whose data cannot mix for compliance reasons. Time-slicing shares GPU compute across workloads, but it does not provide hardware-level memory isolation in the way MIG does. Organizations with strict tenant isolation requirements generally avoid relying on shared GPU memory between unrelated workloads because hardware partitioning provides stronger isolation guarantees.
For genuinely untrusted or regulated workloads, MIG is the safer default, not time-slicing. A MIG partition has its own dedicated memory and fault domain, providing much stronger isolation between workloads than time-slicing. The cost is lower flexibility. Fixed partition sizes can leave GPU memory or compute underutilized when workloads do not fit the available profiles. For regulated workloads, that is usually a fair trade. For an internal tool serving one trusted team, it is often not worth it.
The blunter but more defensible option, especially for anything touching customer data across tenant boundaries, is dedicated node pools per tenant rather than sharing a GPU at all. It costs more in idle capacity. It also removes an entire category of isolation questions from a security review, which matters more than the extra spend once a customer's legal team starts asking how workloads are kept separate. Decide this by trust boundary, not by convenience. A cluster running one company's internal models can share aggressively. A platform serving external customers on the same GPU fleet cannot, no matter how good the utilization numbers look on paper.
The cold start problem nobody warns you about
Model loading is the single most underestimated bottleneck in LLM deployment on Kubernetes. A 70-billion-parameter model stored in FP16 weights occupies roughly 140 GB. The actual GPU memory requirement is higher once runtime overhead, activations, and the KV cache are included. Loading those weights from object storage, a shared volume, or a cached image layer into GPU memory can still take several minutes, depending on storage throughput and network performance. If a pod restarts during a traffic spike, that is several minutes of degraded capacity at exactly the wrong moment.
A few practical fixes matter more than most tuning:
- If you package model weights into the container image, use a dedicated image layer and node-level image caching so a restart on the same node does not re-pull the weights. When weights are stored externally, use a shared high-throughput volume or object storage with local caching.
- Use local caching for externally stored model weights so cold nodes avoid repeatedly downloading the same model from the origin storage.
- Separate readiness from liveness carefully. A pod still loading weights should report not-ready, not get killed and restarted, which only resets the clock and wastes the work already done.
This is where storage architecture decisions, usually treated as a boring infrastructure detail, directly determine whether an autoscaling event actually helps during a traffic spike or just adds latency while new pods warm up.
Choosing a serving layer
Kubernetes does not serve model requests on its own. A serving layer sits between the API and the raw model process, and the choice affects latency, batching efficiency, and how much custom code your team has to maintain.
| Framework | Best fit | Real tradeoff |
| KServe | Teams already standardized on Kubernetes-native ML tooling | Strong standardization and multi-framework support, but adds an abstraction layer that can obscure GPU-level tuning |
| vLLM | High-throughput LLM serving with continuous batching | Excellent throughput and memory efficiency for transformer models, but less suited to non-LLM model types |
| NVIDIA Triton | Mixed fleets serving LLMs alongside traditional ML models | Flexible across model formats, but has a steeper learning curve |
| Ray Serve | Teams building custom Python inference pipelines and multi-step AI applications | Excellent orchestration, but operating a Ray cluster adds complexity |
Most teams do not need to pick just one. A common and reasonable pattern is vLLM for the core LLM serving path, wrapped by KServe for the Kubernetes-native routing, scaling, and canary rollout tooling. Adding Ray Serve on top only makes sense once the workload genuinely needs multi-step orchestration, not just inference.
Autoscaling generative workloads without wrecking latency
The Horizontal Pod Autoscaler scales on CPU or memory by default. Neither metric means much for a GPU-bound inference pod, which can sit at low CPU utilization while its GPU is fully saturated and its request queue is growing. Scaling on GPU utilization directly is better, but still incomplete. A single long-running generation request can hold a GPU busy without reflecting queue pressure at all.
KEDA can scale on external metrics such as queue depth or requests-in-flight, provided those metrics are exposed through systems such as Prometheus, Kafka, or another supported scaler. These metrics usually correlate much more closely with user-facing latency than CPU utilization does. The tradeoff is that GPU nodes are slow to provision. Even with a correct scaling signal, a new node with a fresh GPU can take several minutes to become schedulable, and the cold start problem above adds more time on top of that. Autoscaling smooths out sustained load. It does not fix a sudden five-times traffic spike, and treating it as if it will is a common and expensive mistake.
Keeping a small buffer of warm, pre-loaded pods above the strict minimum needed for average traffic is not elegant, and it costs real money sitting idle. It is also often the only thing that keeps latency acceptable during a spike, because no autoscaler can outrun a multi-minute cold start.
Observability that actually catches GPU problems
Standard Kubernetes dashboards, built around CPU, memory, and pod restarts, miss almost everything that matters for a GPU workload. A pod can look perfectly healthy on every metric Kubernetes tracks by default while its GPU sits at high memory utilization and fragmentation that will crash the next request.
Three signals are worth wiring up before the first incident, not after:
- GPU memory utilization, GPU utilization, and ECC or hardware health metrics, collected through DCGM Exporter rather than inferred from pod-level memory requests.
- Queue depth and time-to-first-token, which tell you when users are waiting even if every pod reports as healthy.
- Per-model request latency, broken out individually if a cluster serves more than one model. An aggregate latency number hides a single slow model behind several fast ones.
None of this is exotic tooling. It is mostly a matter of deciding to wire it up before the first 2 a.m. page, rather than after.
Where this gets expensive
GPU capacity is the most expensive line item in most generative AI infrastructure budgets, and Kubernetes does not make that go away. Running GPU nodes on spot or preemptible instances cuts cost significantly, sometimes by more than half, but it introduces real risk. A preempted node mid-inference drops in-flight requests, and a model that takes minutes to reload on a fresh node makes that preemption expensive in a different way. Spot capacity works well for batch inference and asynchronous workloads. It is a harder sell for latency-sensitive, user-facing chat or agent workloads, where a multi-minute recovery window is not acceptable.
The honest tradeoff is this: full GPU isolation with on-demand capacity gives predictable latency and a predictable, high bill. Aggressive sharing and spot capacity gives a much lower bill and real risk of latency spikes or dropped requests during preemption events. Most teams land somewhere in the middle, running steady baseline traffic on reserved or on-demand GPUs and handling burst or batch workloads on spot capacity. That split has to be designed deliberately. It does not happen by default.
Takeaway
Kubernetes can run LLM and generative AI workloads well, but the parts that make it work are not the parts most guides cover. GPU scheduling strategy, model loading architecture, and queue-aware autoscaling matter more than which YAML template you start from. Treat GPU capacity planning and model loading as first-class design problems before you write a single manifest, and the serving-layer and autoscaling choices become much easier to get right.
Frequently Asked Questions (FAQs)
Q1: Is Kubernetes actually a good fit for running LLMs?
Yes, for most teams already running other workloads on Kubernetes. It gives you a consistent operational model across services. The caveat is that default Kubernetes patterns, built for stateless web services, need real adjustment for GPU scheduling, model loading, and autoscaling before an LLM workload behaves reliably in production.
Q2: Do I need MIG or is time-slicing enough for GPU sharing?
It depends on how many tenants share a cluster and how much isolation they need. Time-slicing is simpler and works for a small number of trusted workloads. MIG adds real memory isolation and is worth the extra configuration once multiple teams or customers share the same GPU fleet.
Q3: How long does it actually take to load a large model on a cold pod?
It varies with model size, precision, and storage throughput, but a large model commonly takes several minutes to load from storage and into GPU memory during a cold start. This is why model caching and readiness probe design matter as much as autoscaling configuration.
Q4: Should I use KServe, vLLM, Triton, or Ray Serve?
Most teams do not need to choose only one. vLLM handles the core LLM inference path efficiently, and KServe adds Kubernetes-native routing and rollout tooling on top. Triton fits better for mixed fleets serving several model types. Ray Serve is worth the added operational weight only when the workload needs genuine multi-step orchestration, not just inference.
Q5: Are spot instances safe for production LLM inference?
They work well for batch and asynchronous workloads, where a preempted node just delays a job. For latency-sensitive, user-facing inference, preemption mid-request combined with a multi-minute cold start on the replacement node is a real risk. Most production setups keep baseline traffic on on-demand or reserved capacity and reserve spot capacity for burst or batch work.
