Context
A server can report a full filesystem while a directory listing seems far too small to explain it. The apparent mismatch often comes from measuring different things: allocated filesystem space, visible file paths, inode availability, or data still held by an open process after a filename was removed. The right response begins with the exact path where writes fail.
Consider an illustrative web server whose large access log was deleted during an incident. The file disappears from the directory, but free space does not return. The web server still holds the file open. Deleting more unrelated files can create unnecessary damage while leaving the main allocation unchanged.
Confirm the filesystem and the kind of exhaustion
GNU df reports filesystem space and can report inode usage. Directory-based tools answer a different question about reachable files. GNU's guidance and the lsof project describe why an unlinked file can continue occupying storage while a process retains an open reference. lsof can help identify those references, subject to platform and permission limits. 123
Run read-only checks against the failing path, not only the root filesystem. Commands such as df -h /var/log and df -i /var/log can identify the relevant mount and whether blocks or inodes are constrained. Use the actual path from the application error.
Compare measurements with consistent scope
A directory scan may cross into other mounts unless constrained appropriately, while permissions can hide parts of the tree. Container mounts, bind mounts, snapshots, quotas, and reserved filesystem space can also affect the comparison. Record the mount layout before concluding that one tool is wrong.
Use a bounded scan during a busy incident. A full traversal of millions of files can add significant I/O and delay recovery. Start with likely owners such as application logs, temporary uploads, caches, and backup output, then narrow the investigation using evidence. Keep customer filenames out of broadly shared reports.
Look for deleted files still held open
Use a supported lsof query such as lsof +L1 through an appropriately authorized account to inspect unlinked open files. Correlate the reported process, file, and filesystem with the storage pressure. The output can include unrelated deleted libraries or temporary files, so size and ownership matter.
Do not sum every displayed row blindly. Several file descriptors can refer to the same underlying file. Identify the actual allocation and owning process before estimating recoverable space. A planned service action should target the relevant handle lifecycle rather than treating all lsof output as waste.
Choose a service-aware release of space
For a log file held by a service, use the documented reopen or rotation procedure where supported. A controlled restart may be necessary in some cases, but assess request draining and service availability first. Closing the final relevant handle allows the filesystem to reclaim storage according to its normal semantics.
Avoid blindly truncating file descriptors through process paths or deleting database-managed files. Those shortcuts can damage application expectations or remove useful evidence. The operations decision should name the owning service, the data's purpose, and the supported cleanup or restart method.
Investigate inode pressure separately
A filesystem can run out of inodes while substantial byte capacity remains. Workloads generating many small sessions, cache entries, or temporary files can create this pattern. Use inode counts and a safe directory-level investigation to locate the producer.
Cleanup should follow the application's lifecycle rules. Deleting every small file by age can remove active sessions or work in progress. Confirm which files are disposable, whether the application can recreate them, and how concurrent writers behave. Then fix the retention or producer issue so the inode population does not immediately grow again.
Check quotas and the application's own limits
A user, project, container, or application quota can reject writes before the underlying filesystem is full. Read the exact error and inspect the relevant limits. A command run as an administrator may succeed while the service account remains blocked by its quota.
For a container, verify both the mounted data volume and the writable layer or ephemeral-storage budget. The application may write temporary data outside its persistent mount. Increasing the data volume will not help if the failing temporary path remains on a different constrained layer.
Validate recovery with the real writer
After the chosen action, recheck free blocks, free inodes, and the relevant quota. Perform a safe write through the application or service account, then verify normal processing. A successful root-level touch command answers a narrower question than the customer's upload or database transaction.
Observe the growth rate after recovery. If space is reclaimed but fills again quickly, identify the active producer before closing the incident. Preserve a short timeline linking the growth, cleanup, service action, and restored business operation. That makes the next investigation substantially faster.
Turn the incident into a capacity policy
Set retention and rotation rules for the identified data family, and monitor the exact mount and inode usage. Include projected exhaustion when growth is predictable. Verify that log rotation actually causes the service to reopen files under its documented procedure.
A focused operator exercise: An open-log demonstration
In a disposable test filesystem, start a harmless writer, remove the test file's directory entry, and compare the visible directory with filesystem usage while the writer remains active. Use the supported open-file inspection tool to identify the process reference. Then stop or reopen the writer through its intended lifecycle and observe reclaimed space.
This exercise should never use a database data file or important service log. Its purpose is to teach the difference between an absent filename and a released allocation. Keep a second inode-limited test separate if the environment needs it, because many tiny files and one deleted open file require different remediation paths.
Quick diagnostic map
| Evidence | Review | Acceptance boundary |
|---|---|---|
| Block usage | Compare filesystem scope in df and du | Check mount boundaries before deleting files |
| Open handles | Inspect deleted files still held by processes | Release them through a safe service action |
| Inode pressure | Compare inode and byte availability | Fix the actual exhausted resource |
Frequently asked questions
Why did deleting the log not free space? A process may still have the unlinked file open. Inspect its owner and use a supported release procedure.
Clouvenix can trace the failing path through mounts, files, handles, and quotas, helping teams reclaim capacity without deleting the wrong data or masking the producer that caused the incident.
For a scoped review of this workflow, discuss the relevant Clouvenix engineering service.