When a ZFS dataset fails to mount, the directory it should appear under can show whatever is underneath it, and the data in the dataset looks gone. The data is usually still in the pool. The fix is to find out why the dataset is not mounted and to bring it back onto its configured mountpoint, before anyone changes anything else.
What “shadowing” means in ZFS
A ZFS filesystem dataset is mounted at the path set by its mountpoint property. Suppose a dataset named tank/projects has mountpoint=/srv/projects. When it is mounted, its files appear under /srv/projects. When it is not mounted, the directory /srv/projects still exists on the root filesystem, and whatever is stored there, often an empty directory or a few stray files, is what you see. From the shell, the dataset looks deleted. Nothing has been removed from the pool.
As an Amazon Associate I earn from qualifying purchases.
That is the mechanism behind the word “shadowing” in this kind of incident. Missing files at a path do not prove missing data. They prove that the path is not showing the dataset you expected at that moment.
What the title does and does not establish
The headline describes an outage in which about 5.5 TB appeared to be missing. Publicly, that figure is not tied to a named system, a pool layout, or a measurement method, so this article does not claim to know the exact cause of that particular event. It explains the documented behavior that would produce this symptom and how to tell which situation you are in.
#1 Best Overall
- IronWolf internal hard drives are the ideal solution for up to 8-bay, multi-user NAS environments craving powerhouse performance
- Store more and work faster with a NAS-optimized hard drive providing ultra-high capacity up to 16TB and cache of up to 256MB
- Purpose built for NAS enclosures, IronWolf delivers less wear and tear, little to no noise/vibration, no lags or down time, increased file-sharing performance, and much more
- Easily monitor the health of drives using the integrated IronWolf Health Management system and enjoy long-term reliability with 1M hours MTBF
- Three-year limited warranty protection plan included and three year Rescue Data Recovery Services included
Two numbers can look like “5.5 TB hidden” and mean different things. The used property reports space consumed by a dataset and its descendants under ZFS accounting rules, and it reflects compression and snapshots. The logicalused property reports the logical space consumed by the dataset and its descendants, ignoring compression. If a total came from one and was compared against the other, or against the capacity of a different filesystem, the gap can be misread as missing data. Record which property or tool produced any size figure before you act on it.
The conditions that leave a dataset unmounted
Pool import and dataset mounting are separate steps. On boot, OpenZFS imports the pool first and then mounts datasets, in mountpoint order. A dataset can therefore be missing for several different reasons, and each needs a different fix.
The pool is not imported
If the pool itself did not import, none of its datasets can mount. The pool’s contents will not appear in zfs list at all. This is a pool-level problem, and you should check it with zpool status and zpool import before looking at datasets.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Rank #2
- Store more, compute faster, and do it confidently with the proven reliability of BarraCuda internal hard drives
- Build a power house gaming computer or desktop setup with a variety of capacities and form factors
- The go to SATA hard drive solution for nearly every PC application from music to video to photo editing to PC gaming. Ax. Sustained transfer rate OD: 190MB/s
- Confidently rely on internal hard drive technology backed by 20 years of innovation
- Frustration Free Packaging - This is just an anti-static bag. No cables, no box.
The dataset is not mounted, but the pool is healthy
This is the most direct form of shadowing. The pool imported, but the dataset’s filesystem is not attached to its mountpoint. Common causes include a mountpoint directory that did not exist or was not empty when mounting was attempted, a canmount setting that prevents automatic mounting, and a service ordering problem where the dataset mounted before a dependency was ready.
OpenZFS requires that the mountpoint path exist and be empty before a filesystem is mounted there, unless an overlay mount is explicitly requested. If stray files are sitting in the directory, the dataset will refuse to mount over them, and the stray files become what you see.
The dataset is encrypted and its key is not loaded
An encrypted dataset without its key cannot be mounted. Its path will look empty or show the underlying directory, and the data is intact. Check the keystatus property before concluding anything else.
Rank #3
- IronWolf internal hard drives are the ideal solution for up to 8-bay, multi-user NAS environments craving powerhouse performance.date transfer rate:6.0 gigabits_per_second
- Store more and work faster with a NAS-optimized hard drive providing 8TB and cache of up to 256MB
- Purpose built for NAS enclosures, IronWolf delivers less wear and tear, little to no noise/vibration, no lags or down time, increased file-sharing performance, and much more
- Easily monitor the health of drives using the integrated IronWolf Health Management system and enjoy long-term reliability with 1M hours MTBF
- Three-year limited product warranty protection plan and three year Rescue Data Recovery Services included
The mountpoint was changed
Changing a mountpoint property unmounts and remounts the filesystem, and child datasets that inherit the mountpoint move with it. If an administrator changed a parent’s mountpoint during maintenance, the data may have moved to a path nobody was checking. The affected datasets still exist; they are simply at a different location.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCrashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteA triage sequence for a suspected shadowed dataset
Run these checks in order. Do not run any command that writes to the pool, such as zfs set on a mountpoint or zfs mount -O, until you have recorded the current state.
- Confirm the pool is imported. Run
zpool status -xto see the health of imported pools, thenzpool importwith no arguments to list pools available for import but not yet imported. If the pool you expect is in the second list, it is not imported. - Record what is mounted. Run
zfs mountwith no arguments. This lists mounted ZFS filesystems and their mount points. Save the output before changing anything. - List the datasets and their expected paths. Run
zfs list -o name,mountpoint,mounted,canmount,keystatus. Any dataset withmountedset tonowhose data you expected at a path is a candidate for shadowing. - Check the path itself. Run
ls -laon the mountpoint and compare it with the dataset’s contents, usingzfs list -rto confirm the dataset holds the data you expect. - Clear the mountpoint and mount. If the directory is non-empty because of stray files, move them aside, not delete them. Then run
zfs mounton the specific dataset, orzfs mount -ato mount all datasets whose mountpoint andcanmountsettings allow it. - Load keys if needed. For encrypted datasets with
keystatusset tounavailable, load the key withzfs load-keyand then mount.
If a dataset is still not mounted after these steps, the reason will be in the output of zfs mount errors or the system journal from boot, not in the dataset properties.
Rank #4
- Available in capacities ranging from 2 to 22TB(1) | (1) 1GB = 1 billion bytes and 1TB = 1 trillion bytes. Actual user capacity may be less depending on operating environment.
- For RAID-optimized NAS systems with unlimited number of bays
- Rated for 550TB/yr workload rate(2) | (2) Annualized Workload Rate = TB transferred x (8760 / recorded power-on hours). The maximum rated workload is specified for operating at typical temperature of 40C. Workload Rate will vary depending on your hardware and software components and configurations.
- Designed to handle the demands of high-intensity 24x7 multi-user NAS environments
- Western Digital partners with a wide range of NAS system vendors for extensive testing to ensure compatibility with most NAS enclosures
Comparing the possible explanations
The table below separates the situations that produce the same symptom. Each one has a different first check and a different risk.
| Situation | What you see | First check | Data at risk? |
|---|---|---|---|
| Pool not imported | Pool absent from zfs list |
zpool import |
No, if the pool is healthy |
| Dataset not mounted | Path shows empty or stray directory | zfs list -o name,mountpoint,mounted |
No |
| Mountpoint directory not empty | Stray files visible at the path | ls -la on the mountpoint |
No, but moving files blindly is risky |
| Encrypted dataset, key not loaded | Empty or misleading path | keystatus property |
No |
| Mountpoint changed | Data present at a different path | mountpoint on each dataset and its parents |
No |
| Actual data loss | Data absent in the pool and snapshots | zfs list -t snapshot and pool health |
Yes |
Only the last row represents real loss. Every other row is recoverable by correcting mount state, and most are resolved in a few commands.
Reducing the chance of a repeat
- Set an explicit
mountpointon every dataset that holds data your services read, and keep a written inventory of expected paths. - Keep mountpoint directories empty. A stray file in a future mountpoint is the most common way to block a mount.
- Use
zfs list -o name,mountpoint,mountedin post-reboot checks so an unmounted dataset is noticed before users report missing files. - Take snapshots before changing mountpoints or running maintenance, so a misplaced dataset can be compared against a known state.
- Check
keystatusfor encrypted datasets as part of boot verification, since a missing key looks like missing data.
When the data is not where you expect after an outage, assume it is still in the pool until the pool and dataset checks above prove otherwise. Most cases of apparently vanished data in ZFS are mount-state problems, and they can be corrected without restoring from backup.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
