Kubelet does monitor inode availability on Linux, but its default inode limits are hard eviction thresholds—not early-warning triggers for image cleanup. A node can therefore run short of inodes while byte usage still looks comfortable, then reach a point where kubelet has no grace period before acting. The key is to monitor inode and byte use separately and check which filesystem is under pressure.
Why can a Kubernetes node run out of inodes while disk space remains?
Filesystems track both stored data and the files that data occupies. Bytes measure how much space file contents consume; inodes represent filesystem objects such as files and directories. A workload that creates very many small files can exhaust available inodes before it fills the filesystem’s byte capacity.
As an Amazon Associate I earn from qualifying purchases.
A 2026 CNCF article by Onkar S. illustrates the distinction with an ext4 demonstration: an image populated with 4,000 small files used 97.9% of its inodes and 32.6% of its blocks, leaving 84 inodes free. Those figures describe that demonstration, not a prediction for other filesystems or workloads. In the article’s incident account, a dependency directory contained 21,553 files per snapshot and the image contained more than 40,000 files. The author said the cluster was no longer available to recheck the original system captures, so treat those incident figures as author-reported rather than independently verified.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →In that case, the inode-heavy content was found in containerd’s overlayfs snapshot store. Repeated small-file dependency trees and image-layer changes can multiply stored snapshots, but containerd and Node.js are not inherently the cause of inode exhaustion.
#1 Best Overall
What does kubelet monitor, and when does it react?
Kubernetes v1.37 documentation lists these default hard eviction thresholds on Linux. They are configurable, so inspect the node’s actual kubelet configuration before diagnosing a live cluster.
| Signal | Default hard threshold | What it measures |
|---|---|---|
nodefs.inodesFree |
Below 5% | Free inodes reported for the node filesystem |
imagefs.inodesFree |
Below 5% | Free inodes reported for the runtime image filesystem |
nodefs.available |
Below 10% | Available bytes on the node filesystem |
imagefs.available |
Below 15% | Available bytes on the runtime image filesystem |
The inode signals come from node statistics: nodefs.inodesFree uses node.stats.fs.inodesFree, while imagefs.inodesFree uses node.stats.runtime.imagefs.inodesFree. The documentation describes these signals for Linux. A hard eviction threshold has no grace period. The default housekeeping-interval for evaluating eviction thresholds is 10 seconds, but that interval is not a promise of a 10-second buffer after a threshold is crossed.
Rank #2
There is an important configuration trap: when any eviction-hard parameter is customized, other defaults are not automatically inherited unless MergeDefaultEvictionSettings is enabled. Supply or verify the complete intended set of thresholds rather than assuming an override preserves every default.
Why doesn’t image garbage collection prevent inode pressure?
Image garbage collection and inode eviction respond to different signals. The CNCF article reports that kubelet’s imageGCHighThresholdPercent and imageGCLowThresholdPercent defaults concern byte use: garbage collection begins at 85% use and works down to 80%. Verify these values for the Kubernetes version and configuration in use. The process does not serve as an inode-count cleanup trigger, so an image filesystem can cross an inode threshold even when byte use remains below the image-GC trigger.
When kubelet detects DiskPressure, it attempts applicable node-level reclamation before evicting end-user pods. Depending on the filesystem layout, it may garbage-collect dead pods and containers for nodefs or containerfs pressure, and delete unused images for imagefs pressure. If reclamation does not bring the signal below its threshold, kubelet proceeds to pod eviction. For inode or PID starvation, relative pod priority determines eviction order because pods do not request those resources.
What do nodefs, imagefs, and containerfs refer to?
These are kubelet-observed filesystem identifiers, not a guarantee of three separate mounted filesystems. Depending on runtime and layout, node data, images, writable layers, and local ephemeral data may share a filesystem or be split across filesystems. Kubernetes v1.37 documentation describes containerfs for supported configurations; that release requires the KubeletSeparateDiskGC feature gate and identifies CRI-O v1.29 or later as a supporting runtime. This support detail is version-sensitive.
Do not assume kubelet accounts correctly for every mount just because it runs on the node. Kubernetes local ephemeral-storage documentation warns that extra filesystems mounted under paths such as /var/lib/kubelet, /var/log, or runtime storage outside supported layouts can prevent accurate reporting. A tmpfs emptyDir is tracked as memory use rather than local ephemeral storage.
How should you diagnose inode pressure?
- Check bytes and inodes on the affected mount. Run
df -h /anddf -i /as an initial check, then substitute the actual mount used by the affected filesystem. Root may not be the mount under pressure. - Find directories with many inodes. For a containerd store, the CNCF article suggests
du --inodes -xS /var/lib/containerd | sort -rh | head -n 20. Here,-xstays on one filesystem and-Savoids summing descendants into parent directories. Check that your host’sdusupports these options. - Map the result to the runtime and filesystem layout. Confirm which mount corresponds to nodefs, imagefs, or containerfs on the affected node; do not infer that these identifiers map to distinct mounts.
- Inspect image contents and build layers if snapshots dominate. Review whether source trees or development dependencies are copied into runtime images, whether
.dockerignoreexcludes unnecessary files, and whether a multi-stage build can leave build-only files out of the final image. These are mitigation ideas, not a measured before-and-after result. - Use node-side cleanup cautiously. The article notes that
crictl rmi --pruneremoves images not currently used by containers, which might have to be pulled again later. Removing all stopped containers can also remove access to their previous logs. Confirm command behavior for the installed runtime and understand the consequences before cleanup.
How can you catch inode pressure earlier?
Alert on inode consumption separately from byte consumption. The CNCF article gives 80% inode use as an example alert threshold, while explicitly treating it as an operator judgment call—not a Kubernetes default. Choose a limit that leaves enough time to investigate and respond for your workload, filesystem, and recovery process.
A Prometheus query can compare inode use with byte use, but its metric names and labels must match the exporter and mount labels in your environment. For example, with a node exporter exposing the usual filesystem metrics, these expressions show the used percentage for each filesystem:
100 * (1 - node_filesystem_files_free / node_filesystem_files)
100 * (1 - node_filesystem_avail_bytes / node_filesystem_size_bytes)Filter out pseudo-filesystems and select the relevant device or mount before turning either expression into an alert. A filesystem can have modest byte consumption and high inode consumption at the same time; a byte-only dashboard will not reveal that condition.
Which evidence applies to your node?
Kubernetes v1.37 documentation is the reference for the listed default thresholds and behavior; a node’s version, kubelet configuration, runtime, operating system, and filesystem layout can change what applies. The CNCF article provides a useful incident account and ext4 demonstration, but its author could not recheck the original cluster captures. Treat the incident’s file counts as context for a plausible failure mode, not a universal rate or benchmark.
Quick Recap
- Kubernetes: Node-pressure Eviction describes eviction signals, thresholds, reclamation, and filesystem identifiers.
- Kubernetes: Local Ephemeral Storage explains kubelet storage accounting and layout limitations.
- CNCF: Kubelet watches inodes. Just not until it’s an emergency. contains the attributed case account and diagnostic suggestions.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
