Context
Container images share node disk with other runtime responsibilities. When that storage fills, image pulls can fail and kubelet may report garbage-collection problems or node pressure. Recover capacity through the supported kubelet and runtime lifecycle rather than deleting containerd directories while workloads are running.
Identify the filesystem that is full
Record node conditions, image-pull events, kubelet logs, runtime status, free bytes, and free inodes. Distinguish the node filesystem from image or container filesystems in the supported layout. The path holding application logs may differ from the path holding image content.
Inspect disk growth over time. A large new image, accumulated unused images, container writable layers, logs, and unrelated host files require different corrections. A single free-space percentage cannot identify which population is safe to reclaim.
Understand kubelet's image lifecycle
Kubelet cooperates with the runtime to manage unused images. Its garbage-collection configuration includes high and low disk-usage thresholds, with additional options depending on Kubernetes version. Use the current KubeletConfiguration reference for the deployed cluster.
Kubernetes advises against independent external cleanup tools that can interfere with kubelet's view of containers and images. A generic host cron job that removes runtime files may free bytes while breaking running containers or future image operations.
An image still needed by a running container is different from an unused image eligible for collection. Shared layers and runtime snapshots also complicate a naive sum of image sizes. Use supported inventory and diagnostics rather than equating every listed image size with separately reclaimable physical bytes.
Check why reclaim did not create enough room
Read the specific garbage-collection event and requested versus reclaimed space where available. There may be too little eligible unused content, an active workload consuming writable storage, a filesystem error, or a runtime problem preventing cleanup.
Compare image rollout size with available pull headroom. A node may need space for both the previous and new image during a rollout, plus unpacking and writable layers. Aggressive cleanup followed by an equally aggressive image repull cycle can increase registry traffic and deployment latency.
Review log rotation and workload local-storage limits separately. Image garbage collection is not a general cleanup mechanism for unlimited application logs or temporary exports. Fix the source of growth rather than repeatedly forcing the runtime to compensate.
An illustrative rollout failure
A new application image is substantially larger than its predecessor. Nodes are normally near their accepted storage threshold, and the rollout needs old and new content simultaneously. Image GC runs but cannot reclaim enough unused data, so replacement Pods fail to pull.
The team reduces unnecessary build content, provides suitable node storage headroom, and validates rollout overlap. It keeps rollback artifacts available in the registry and tests pulling them when needed. Deleting arbitrary runtime directories would have endangered the healthy replicas that were preserving service.
Quick diagnostic map
| Evidence | Boundary to test | Next check |
|---|---|---|
| Image GC reclaims less than requested | Eligible content insufficient | Running images and other disk users |
| Bytes available; inode shortage | File-count pressure | Inodes and small-file growth |
| Space fills only during deployment | Rollout overlap | Image size and unpack headroom |
| Repeated runtime errors | Cleanup cannot complete | Supported runtime and filesystem diagnosis |
A focused operator exercise
Use a disposable node pool with the production runtime and image sizes. Measure free bytes and inodes before pulling a candidate image, during unpacking, during rollout overlap, and after normal garbage collection. Include logs and expected temporary work in the test.
Exercise the supported node replacement or maintenance procedure when capacity is insufficient. Verify active workloads remain available according to the platform's disruption design. Confirm the previous image can be repulled for rollback and record how long that takes. The receipt should explain both reclaimed space and required future headroom.
Prevent recurrence through image and node policy
Track image-size changes during build review and alert on node storage growth with enough time for an ordinary maintenance response. Keep kubelet thresholds, node disk sizing, and log policy aligned.
When changing runtime or filesystem layout, revalidate metric collection and cleanup behavior. An operating procedure accepted on one layout may not apply unchanged to another.
Frequently asked questions
Should I delete containerd's content directory to free space?
Use the supported runtime and node recovery procedure. Direct deletion can invalidate running workloads and runtime bookkeeping.
Will image GC remove application logs?
It manages eligible image content. Diagnose logs, writable layers, and temporary files through their own lifecycle and capacity controls.
Continue the investigation
- Docker Disk Growth: Separate Layers, Logs and Volumes
- Kubernetes Ephemeral Storage Evictions: Account for Logs and emptyDir
Browse the cloud, server and infrastructure guide library for related operational questions.
Explore Clouvenix Containers & Kubernetes to discuss implementation, validation and operational handover.