← All insights
ENGINEERING JOURNAL / Azure Operations

AKS Pod IP Exhaustion: Inspect the Allocation Mode

Investigate AKS pods stuck during sandbox creation by checking the Azure CNI allocation mode, pod subnet capacity and node address requests before recycling workloads.

AKS Pod IP Exhaustion: Inspect the Allocation ModeAKS Pod IP Exhaustion: Inspect the Allocation Mode — conceptual 3D-style vector showing Pending pod, Node allocator, Pod subnet, Address batch, Created sandbox.. Conceptual teaching illustration; connections show the investigation flow rather than a vendor deployment topology.CLOUVENIX / AZURE OPERATIONSAKS Pod IP Exhaustion: Inspect the Allocation Mode01Pending pod02Node allocator03Pod subnet04Address batch05Created sandboxConceptual investigation flow / AKS Pod IP Exhaustion: Inspect the Allocation Mode
AKS Pod IP Exhaustion: Inspect the Allocation Mode — conceptual 3D-style vector showing Pending pod, Node allocator, Pod subnet, Address batch, Created sandbox.

Context

A Kubernetes pod needs a network sandbox before its containers can start. If Azure CNI cannot allocate an address, the pod can remain in ContainerCreating even when CPU and memory are available. Restarting the application container cannot supply missing subnet capacity because the failure happens earlier.

Prerequisites are a failing pod's events, its assigned node and the cluster's network configuration. Determine whether the cluster uses traditional Azure CNI, a separate pod subnet with dynamic or static block allocation, or another supported mode. Capacity calculations must match that choice.

1. Confirm an address allocation failure

Inspect the affected pod's description and recent events. Look for FailedCreatePodSandBox with explicit allocation or unavailable-address details. An image pull error or volume attachment failure can produce a similar waiting state but needs a different investigation.

Record the node, pool and incident time. Compare new pods with already running pods on that node. Existing traffic can remain healthy while new sandboxes fail. Preserve events before repeated deletion replaces the evidence with another identical failed startup.

2. Inspect the correct subnet and allocator

Read the pool's node and pod subnet identifiers. In separate pod subnet modes, the free address count in the node subnet does not answer whether pods have capacity. Include other pools or clusters sharing the pod subnet when calculating demand.

Dynamic mode acquires addresses in batches, so running pod count is not the whole allocation picture. Use the supported read-only node network configuration view, such as kubectl get nnc -n kube-system -o wide for the relevant dynamic setup. Compare requested and allocated addresses with current pod demand. Static block allocation instead reserves blocks according to its configuration and needs its own sizing calculation.

3. Distinguish exhaustion from delayed reclamation

Check whether scaling, surge nodes or rapid pod churn preceded the failure. Address requests can outpace reclamation, and allocator problems can leave unexpected allocations. Compare pool activity, subnet usage and supported CNI diagnostics before concluding that all allocated addresses belong to active pods.

For genuine capacity shortage, review a supported expansion or new-pool plan using an appropriately sized subnet. For apparent allocator leakage, preserve diagnostics and follow current AKS support guidance. Do not manually remove host CNI state files or detach addresses from running nodes as an improvised cleanup procedure.

Diagnostic checkpoints

ObservationLikely boundaryNext evidence
Sandbox event says no addressesCNI allocationPool networking mode
Node subnet free, pod subnet fullSeparate capacity poolsPod subnet consumers
Allocation far exceeds running podsBatch reserve or reclamationNode network configuration
Failure follows upgrade surgeTemporary address demandConcurrent node operations

Illustrative scenario

A hypothetical team sizes a pod subnet using current replica count but adds two node pools sharing it. A deployment burst cannot obtain another address batch. Existing pods keep working, leading operators to suspect the image. Reading sandbox events and shared subnet consumers reveals the actual boundary. This is an illustrative case without invented utilization measurements.

Verification and networking SOP

After the reviewed correction, create an approved representative workload and confirm sandbox creation, pod IP assignment and readiness. Check several nodes, including the previously affected node. Successful scheduling alone does not establish network setup completion.

Maintain networking mode, shared subnet ownership and peak address assumptions in the SOP. Monitor pending sandbox events alongside available address capacity. Recalculate before changing maxPods, adding pools or increasing upgrade surge. Keep headroom for system pods and temporary operations rather than sizing exclusively for normal application replicas.

Explore Clouvenix containers kubernetes for implementation and operational handover.

Technical reference

Pod subnet concepts and network diagnostics.