October DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsPC HealthRecommendedCrashes, freezes, slowdowns? Check your PC nowSpot repairable issues before they interrupt work.Check PCOctober DealsAmazon USDeal season is back - check today's better picksAmazon US: current deals, useful picks and tech finds.See Picks×
Skip to content
World desk4 min

Why Healthy Disk Usage Can Hide an Inode Exhaustion Failure

A node can have bytes available yet fail to create files when its filesystem runs out of inodes. Here’s how a CI image full of small files exposed the gap.

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

A node can report 58% of its disk space used and still fail to create files because byte capacity and inode capacity are separate filesystem resources. In Sergey Shinder’s reported CI incident, new pods failed with “no space left on device” while the byte-use panel looked healthy; df -i showed the root filesystem’s inodes exhausted. Kubernetes documents inode-free eviction signals separately from available-byte signals, so monitoring bytes alone can miss this failure mode.

How a filesystem can have bytes left but no room for another file

Filesystems track more than the amount of data they can store. They also track inodes: records needed to represent files and directories. A filesystem can therefore have substantial byte capacity remaining while lacking the inode capacity needed to create another file. Many small files can consume inodes while using relatively few bytes.

As an Amazon Associate I earn from qualifying purchases.

In Shinder’s account, the CI nodes’ disk panel correctly showed 58% byte usage, but the root volume had reached 100% inode usage. The error reported to workloads was “no space left on device.” These are incident details from Shinder’s account; the source page’s indexed excerpt does not identify a publication year.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

What happened on the CI nodes

Shinder describes a frontend build image containing node_modules and roughly 400,000 mostly small files. The image was rebuilt several times a day, and nodes unpacked new versions when pulling them. Cleanup was triggered by byte usage at 85%, so the small files could drive inode exhaustion without reaching that byte threshold. One node reportedly retained 43 versions of the image.

#1 Best Overall
Sale
Seagate 2TB Portable Hard Drive | USB 3.0 (STGX2000400)
  • Easily store and access 2TB to content on the go with the Seagate Portable Drive, a USB external hard drive
  • Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
  • To get set up, connect the portable hard drive to a computer for automatic recognition no software required
  • This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
  • The available storage capacity may vary.

As pods failed and moved to other nodes, those nodes pulled images too. By the afternoon, Shinder says 9 of 20 nodes were refusing work. These are the author’s reported counts, not general Kubernetes statistics or independently validated study results.

Why Kubernetes inode settings matter

Kubernetes’ node-pressure eviction documentation lists Linux inode-free signals separately from byte-availability signals. The relevant signals include nodefs.inodesFree and imagefs.inodesFree, alongside byte signals for those filesystems. The documented default hard threshold is 5% free inodes for both nodefs and imagefs on Linux.

Rank #2
Seagate Portable 5TB External Hard Drive HDD – USB 3.0 for PC, Mac, PS4, & Xbox - 1-Year Rescue Service (STGX5000400), Black
  • Easily store and access 5TB of content on the go with the Seagate portable drive, a USB external hard Drive
  • Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
  • To get set up, connect the portable hard drive to a computer for automatic recognition software required
  • This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
  • The available storage capacity may vary.

Those defaults apply only when no threshold parameters are changed. Kubernetes documentation warns that changing a parameter does not automatically preserve the other defaults unless default merging is enabled or the desired thresholds are explicitly configured. Shinder reports that old custom kubelet thresholds had replaced the complete default set, leaving free inodes unwatched. Operators should confirm the behavior for their deployed Kubernetes release rather than assuming a customized configuration inherits every default.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Check the filesystem layout before changing thresholds

Kubernetes documents layouts where nodefs, imagefs, and, where present, containerfs may map to the same underlying filesystem or to distinct filesystems. That affects which resource a signal describes and where image and writable-layer data consume capacity. Do not assume every cluster stores image data in the same place as node files or container writable layers.

Rank #3
Seagate Portable 1TB External Hard Drive HDD – USB 3.0 for PC, Mac, PlayStation, & Xbox, 1-Year Rescue Service (STGX1000400) , Black
  • Easily store and access 1TB to content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
  • Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop. Reformatting may be required for Mac
  • To get set up, connect the portable hard drive to a computer for automatic recognition no software required
  • This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
  • The available storage capacity may vary.

How to investigate an inode-related “no space left” failure

  1. Check bytes and inodes separately. Compare ordinary byte-usage information with df -i on the affected node. A filesystem showing available bytes but 100% inode use points to a file-count constraint rather than ordinary byte exhaustion.
  2. Identify which filesystem is full. Establish whether the affected path is on the node filesystem, image filesystem, or another mounted filesystem. Kubernetes’ signal names depend on the node’s filesystem layout.
  3. Inspect file-heavy workloads and image versions. Look for images that package large dependency trees, caches, generated files, or many small files, and check how many old image versions remain on nodes.
  4. Review kubelet eviction configuration. Verify that the intended free-inode thresholds are explicitly present or that the deployed release’s default-merging behavior is configured as expected. Include the corresponding byte thresholds that the cluster requires.
  5. Check monitoring and cleanup policies. Make sure inode use or free-inode signals are visible and alerted on, and determine whether cleanup depends only on byte-use thresholds.

What Shinder says changed

Shinder describes a bundle of changes rather than a controlled comparison of individual fixes:

  • The node image was changed to format the container volume as XFS; the author says this allocates inodes as needed. That filesystem claim and its suitability for a particular deployment are not established by the Kubernetes documentation cited here.
  • Node panels and matching alerts were updated to show inode use alongside byte use.
  • Kubelet configuration was changed to specify each eviction threshold explicitly, including free-inode thresholds.
  • The frontend image stopped shipping node_modules; dependencies were obtained from a cache volume at job time. Shinder reports that the image’s file count fell from about 400,000 to fewer than 9,000.
  • Unused images were removed after three days regardless of byte usage.

These are the author’s reported remedies, not universal prescriptions. Filesystem choice, image delivery, cache design, kubelet settings, and cleanup rules all depend on the cluster’s storage layout and operational requirements. The cited sources do not provide a benchmark comparing these approaches.

Rank #4
Seagate Portable 4TB External Hard Drive HDD – USB 3.0, 1-Year Rescue
  • Easily store and access 4TB of content on the go with the Seagate Portable Drive, a USB external hard drive.Specific uses: Personal
  • Designed to work with Windows or Mac computers, this external hard drive makes backup a snap just drag and drop
  • To get set up, connect the portable hard drive to a computer for automatic recognition no software required
  • This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable
  • The available storage capacity may vary.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

The practical lesson for node capacity planning

Disk capacity is not a single number. Byte utilization, inode availability, image retention, and the number of files in workloads all affect whether a node can continue creating files. Shinder’s account illustrates why a byte-only dashboard and byte-triggered cleanup policy can overlook a node approaching inode exhaustion. Kubernetes’ separate inode signals provide a way to make that resource visible and include it in eviction planning.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Quick Recap

SaleBestseller No. 1
Seagate 2TB Portable Hard Drive | USB 3.0 (STGX2000400)
Seagate 2TB Portable Hard Drive | USB 3.0 (STGX2000400)
This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable; The available storage capacity may vary.
$129.99
Bestseller No. 2
Seagate Portable 5TB External Hard Drive HDD – USB 3.0 for PC, Mac, PS4, & Xbox - 1-Year Rescue Service (STGX5000400), Black
Seagate Portable 5TB External Hard Drive HDD – USB 3.0 for PC, Mac, PS4, & Xbox - 1-Year Rescue Service (STGX5000400), Black
This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable; The available storage capacity may vary.
$229.99
Bestseller No. 3
Seagate Portable 1TB External Hard Drive HDD – USB 3.0 for PC, Mac, PlayStation, & Xbox, 1-Year Rescue Service (STGX1000400) , Black
Seagate Portable 1TB External Hard Drive HDD – USB 3.0 for PC, Mac, PlayStation, & Xbox, 1-Year Rescue Service (STGX1000400) , Black
This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable; The available storage capacity may vary.
$119.80
Bestseller No. 4
Seagate Portable 4TB External Hard Drive HDD – USB 3.0, 1-Year Rescue
Seagate Portable 4TB External Hard Drive HDD – USB 3.0, 1-Year Rescue
This USB drive provides plug and play simplicity with the included 18 inch USB 3.0 cable; The available storage capacity may vary.
$208.99
Best Value
Sale
UnionSine 500GB Ultra Slim Portable External Hard Drive HDD-USB 3.0
  • [Upgraded Version] - This external hard drive features a mirrored logo stripe combined with a striped anti-slip design, and the rounded corners of the casing make it easier to grip. The stripes also have a heat dissipation function, ensuring stable and fast data transfer.
  • 【Ultra-thin and quiet】 - The motherboard adopts JMicron 578 noise-free solution, giving you a quiet working environment. Lightweight and portable size designed to fit in your pocket for easy portability.
  • 【Ultra-Fast Data Transfers】 - Pairing this external hard drive with JMicron 578 solution USB 3.0 and USB 2.0 interfaces enables blazing-fast data transfer. It boasts theoretical read speeds of up to 125MB/s and write speeds of up to 103MB/s.
  • 【Plug and Play】 - With no software to install, just plug it in and the drive is ready to use.The hard disk chip is wrapped with an aluminum anti-interference layer to increase heat dissipation and protect data.
  • 【What You Get】 - 1 x Portable Hard Drive, 1 x USB 3.0 Cable, 1 x User Manual, Gift-type shell packaging ,Three-year manufacturer's warranty and free technical support services.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
PC Slower Than It Used to Be?Free scan - under a minute
Crashes, No Sound, or Screen Glitches?Free driver scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.