Context
A Pod can be evicted for local storage pressure even when its persistent volume has plenty of free space. Container writable layers, logs, and disk-backed emptyDir volumes consume node-local storage. Diagnose the eviction message and filesystem layout before increasing a PVC or restarting the same workload onto another nearly full node.
Read the eviction reason first
Capture Pod status, events, node conditions, container restarts, and the time of failure. Distinguish a Pod exceeding its applicable ephemeral-storage limit from node-wide DiskPressure. The corrective owner and capacity calculation can differ.
Useful read-only starting points include:
kubectl describe pod POD -n NAMESPACE
kubectl describe node NODE
kubectl get events -n NAMESPACE --sort-by=.lastTimestamp
Replace the placeholders and retain only relevant output. Events are time-limited evidence, so collect them before repeatedly deleting failed Pods. A fresh healthy replica does not explain why its predecessor was evicted.
Inventory every local writer
Inspect application temporary files, upload staging, generated reports, cache directories, access logs, sidecar output, and writable image layers. Include jobs that overlap with ordinary service traffic. A single large export can change the peak storage requirement without changing the application's usual request rate.
Distinguish disk-backed emptyDir from memory-backed emptyDir. Kubernetes accounts memory-backed volumes as memory use rather than ordinary local ephemeral storage. Moving temporary files to memory therefore changes the failure mode and does not make their capacity requirement disappear.
Also identify where container images and logs live on the node. Supported filesystem layouts and runtime capabilities affect what kubelet can measure. Do not assume a custom mount arrangement is accounted for identically to the documented node layout.
Connect requests, limits, and node capacity
Ephemeral-storage requests inform scheduling, while applicable limits can lead to eviction when exceeded. Inspect the admitted Pod specification and any defaults from LimitRange or policy. The manifest in source control may not include every value applied at admission.
Reserve room for node services, images, logs, and reclaim activity. Scheduling only by CPU and memory can place many storage-heavy Pods on the same node. A generous per-Pod limit is not a reservation of that much physical disk unless the request and placement model provide the intended headroom.
Node-pressure eviction is distinct from an API-initiated eviction. Do not assume a PodDisruptionBudget protects a workload from running the node out of disk or guarantees its usual termination grace under pressure.
An illustrative export failure
A reporting Pod writes intermediate archives into emptyDir before uploading the final result. The output store is healthy and its PVC is mostly empty, but three overlapping exports fill local storage. New replicas repeat the same work and the same failure.
The team measures temporary bytes per export, limits concurrency, streams appropriate output, and provides deliberate local headroom. It keeps incomplete work recoverable and ensures cleanup handles cancellation. Increasing the unrelated PVC would not address the local write path.
Quick diagnostic map
| Evidence | Boundary | Next check |
|---|---|---|
| Pod limit exceeded; node otherwise healthy | Workload budget | Local writers and admitted limits |
| Several Pods evicted on one node | Node-wide pressure | Images, logs, capacity, and reservations |
| PVC free; local disk full | Different storage path | Writable layer and emptyDir |
| Memory OOM after using tmpfs | Capacity moved to memory | Memory-backed volume accounting |
A focused operator exercise
Run a representative export on a test node with observed local-storage metrics. Record peak temporary bytes, log growth, concurrent tasks, and cleanup behavior. Cancel the job midway and verify partial files are reclaimed without removing another task's data.
Test the intended request, limit, and concurrency settings with a bounded peak. Confirm the service either completes or rejects excess work clearly before exhausting the node. Verify important state survives rescheduling in a suitable durable store; local ephemeral storage should not be the only copy of accepted business data.
Verify the operating result
Observe multiple job cycles and log rotation after rollout. Alert on growth rate and remaining headroom with an actionable owner. Include local-storage requirements in workload reviews and node-image changes.
Frequently asked questions
Will increasing a PVC fix every storage eviction?
No. Identify the filesystem that reached its limit. Local writable layers, logs, and emptyDir can be independent of the PVC.
Does a PodDisruptionBudget stop disk-pressure eviction?
No. Node-pressure eviction has different behavior. Prevent local resource starvation and design the workload to recover its durable state.
Continue the investigation
- Linux Disk Full but Files Look Small: Check Inodes, Mounts, and Deleted Open Files
- Quota or LimitRange: Find Why Kubernetes Rejects a Pod
Browse the cloud, server and infrastructure guide library for related operational questions.
Explore Clouvenix Containers & Kubernetes to discuss implementation, validation and operational handover.