CloudOptimo Blog

AI workloads have quickly moved from experimental batch jobs to business-critical systems. The infrastructure supporting them has had to mature just as fast. Distributed training on Kubernetes, large language model serving under tight latency requirements, and multi-tenant accelerator clusters shared across engineering teams all present orchestration challenges that general-purpose systems were not built to handle.

Kubernetes failures rarely look dramatic at first.A deployment succeeds, pods appear to be running, and then production traffic slowly starts failing.Some containers restart repeatedly. New pods stay stuck in `Pending`. Logs show nothing useful. The actual issue is often a small configuration mistake somewhere in the cluster.These situations are common in production environments. The difficult part is not always fixing the problem it is knowing where to look first.

A CrashLoopBackOff alert triggers, and a quick inspection of the workload reveals the reason: OOMKilled. In many cases, engineers note that they recently increased memory limits, and observability tools often show utilization hovering at safe levels just moments before the crash. When the metrics do not align with the failure, simply doubling the limits again is a guess rather than a systematic fix.

Modern applications are commonly built using microservices running inside Kubernetes clusters. These services are distributed across multiple pods that scale dynamically based on traffic and workload demands. As applications grow, managing external access to these services becomes difficult. Exposing every service directly to the internet increases infrastructure costs, complicates DNS management, and creates security risks.

This blog explains how Pod-to-Pod communication works inside Kubernetes, starting from localhost communication within a Pod to networking across multiple cluster nodes. It covers how Kubernetes assigns unique IPs to Pods, how CNI plugins enable flat networking, and how Services solve the instability of changing Pod IPs. The article also explains how CoreDNS enables service discovery using stable DNS names and how kube-proxy routes traffic dynamically to healthy Pods. Finally, it explores Network Policies, Ingress controllers, and external traffic flow to show the complete Kubernetes communication architecture from internal Pods to external users

It is 3:00 AM. A critical transaction microservice running on Amazon EKS has just fired a PagerDuty alert. Pods are stuck in Pending, and the scheduler logs show a brutal error: "Failed to create pod sandbox: failed to set up sandbox container network." Node CPU and memory? Completely fine. The real culprit is the VPC subnet. It has run out of private IP addresses. The AWS VPC CNI plugin silently exhausted every available address long before anyone noticed.

While setting up a single-node GPU workload with Kubernetes Dynamic Resource Allocation (DRA) is relatively straightforward, the real challenge begins when transitioning to production at scale. In a single-node sandbox, the primitives are clean, the YAML behaves predictably, and a working ResourceClaim can typically be established in a single afternoon. However, when workloads scale across multiple nodes, distributed training jobs demand physical interconnect affinity, autoscaling models must align with pre-allocated capacity blocks, and nodes must gracefully handle mid-job GPU failures without disrupting other co-located workloads.

Modern AI infrastructure runs on a quiet contradiction. Organizations spend hundreds of thousands of dollars on NVIDIA A100 and H100 accelerators, provision them into Kubernetes clusters, and watch the majority of that investment sit idle — not because of poor planning, but because of how Kubernetes was architecturally designed to think about hardware. A node reports its GPU as "in use," the scheduler marks the resource as allocated, yet actual utilization tells a different story entirely.

Modern applications run across distributed cloud environments using containers, microservices, APIs, and automated deployment pipelines. While containers improved application portability and consistency, managing large-scale containerized workloads introduced new operational challenges around scaling, networking, monitoring, and failure recovery.