A disk sends you back a block that is not the block you wrote. Not often. Perhaps never on the machine under your desk, in the whole time you own it. But NetApp went and counted properly, across 1.53 million drives over 41 months, and found more than 400,000 checksum mismatches. Their paper puts the awkward part plainly: eight per cent of those were discovered during RAID reconstruction, which is to say at the exact moment when the redundancy that was meant to save you has already been spent.
CERN ran their own numbers in early 2007, writing a couple of gigabytes of known bit patterns across the computer centre and reading them back to see what came home changed. Corruption rates around the 10⁻⁷ mark, with awkward clustering, and a recommendation that checksums be implemented and deployed everywhere.
What both studies describe is silent data corruption: a block that comes back changed while every component involved reports success. Everywhere is a big word. Most of the storage stack in common use cannot act on that recommendation, and the reason is architectural rather than lazy.
The Stack That Cannot Ask Itself a Question
Take the arrangement that most Linux machines run: mdadm underneath for the redundancy, LVM above it for the flexible carving-up of space, ext4 or XFS on top for the actual files. Three mature pieces of software, each excellent at its own job, each maintained by people who know precisely what they are doing.
Now put a mirror in front of that and flip one bit on one of the two drives. Nothing dramatic: a cosmic ray, a firmware bug, a controller having a bad afternoon.
You read the file. mdadm serves you one of the two copies, because as far as mdadm is concerned there is nothing to decide. It has two blocks. Both are readable. Neither drive reported an error, because from the drive's point of view nothing went wrong: it stored what it was given, or it thinks it did, and its own error correction saw nothing worth mentioning.
If you scrub that array, mdadm will notice the two copies disagree. It cannot tell you which one is right. It has no way to know. RAID was designed against the failure it could see coming in 1988, which was a drive dying, and against that failure it works beautifully. Against a drive that lies quietly it has no instrument at all.
The filesystem above has the same difficulty from the other direction. ext4 checksums its own metadata, which is genuinely worth having, and it does not checksum your file contents. Even if it did, it would have nowhere to go with the answer, because the copy that might be correct lives one floor down in a component ext4 cannot address. It would be able to tell you that your data is wrong, and then have to sit there.
Three good components, and between them a question nobody in the building is in a position to answer: of these two blocks, which is the one I wrote?
What Happens When You Stop Drawing the Line
ZFS was built at Sun by Jeff Bonwick and Matthew Ahrens, and its founding move was to refuse the division of labour. Not a filesystem that talks to a volume manager particularly well. One thing, which does both, and can therefore know both.
Once that decision is taken, the interesting part is how little else needs deciding. Four capabilities fall out of it, and none of them required a separate act of invention.
The checksum lives with the parent, not with the data. ZFS stores every block's checksum in the block that points to it, so the pointer and the verification travel together all the way up to a root that is itself checksummed. The whole pool is a Merkle tree. When a block comes back from the disk, ZFS already knows what it should have been, and it knew before it asked.
Self-healing needs nothing further. Because the same program holds both the checksum and the redundancy, the failure described above simply does not survive. The bad copy fails verification, the good copy passes, the good copy is returned to your application, and the bad copy is rewritten from the good one while you wait. You are not told, unless you go and look at the pool's error counters. Nothing in the stack had to be coordinated, because there was no stack to coordinate.
Snapshots cost approximately nothing. ZFS never overwrites a live block; a change is written elsewhere and the pointers are updated. Copy-on-write is there for consistency reasons, so that a power cut can only ever leave you with the old state or the new one. But once you have it, a snapshot is a matter of not freeing the old blocks, and that is close enough to free that people take them every fifteen minutes without thinking about it. Compare an LVM snapshot, which needs a size decided up front, degrades write performance while it exists, and breaks when it fills.
Replication knows what changed without looking. This is the one that catches people out. zfs send -i between two snapshots produces the differences, because the filesystem recorded which blocks were touched in which transaction group and can simply read that back. rsync solves the same problem by walking the entire tree, stat-ing everything, hashing what looks suspicious, and asking the far end a great many questions. On ten terabytes of mostly-unchanged data, one of these approaches reads ten terabytes and the other reads a list.
Four features. One decision. That is the whole argument, and everything else in this piece is either evidence for it or fairness towards the alternative.
Where FreeBSD Comes Into It
ZFS arrived in FreeBSD 7.0 in 2008, and the release notes are charmingly cautious about it: experimental support for Sun's ZFS filesystem. By 8.0 the notes simply record that ZFS is no longer in experimental status, and that is the last fuss anyone made about it. Since then it has been part of the base system, built from the same tree as the kernel, released on the same day, documented in the same handbook.
That sounds like a detail and it changes the daily experience considerably.
Because ZFS is licensed under the CDDL, it cannot be merged into the Linux kernel, whose licence does not permit it. So on Linux, OpenZFS lives outside the tree and is rebuilt against each new kernel through DKMS. It works, and a great many people run it happily in production, and it is nonetheless a moving part that can be broken by an upgrade you did not think was about storage. Nobody enjoys that particular palaver at two in the morning.
On FreeBSD the root filesystem is ZFS by default from the installer onwards, the boot loader understands the pool, and bectl gives you a boot environment as a first-class feature: take a snapshot of the running system, upgrade, and if the upgrade is a disaster, select the previous environment at the boot menu and be back where you started. The equivalent on Linux exists in various third-party forms, none of them in the base system, and one prominent attempt at automating it was abandoned by its own vendor in 2021.
None of this is because FreeBSD engineers are cleverer. It is because a project that ships one tree can integrate a filesystem into the boot process, and a project assembled from independently governed parts has to negotiate the same outcome across several of them.
Version Control That Never Heard of Files
Here is where the theory turns into a way of working, and it is the part I use daily rather than admire from a distance.
Once snapshots are effectively free and the filesystem can tell you what changed between any two of them, you have most of the verbs a version control system offers, sitting one level below the files instead of inside them.
git commit -m "before the risky bit" -> zfs snapshot tank/work@before
git diff --name-status -> zfs diff tank/work@before
git reset --hard -> zfs rollback tank/work@before
git branch experiment -> zfs clone tank/work@before tank/try
git log --oneline -> zfs list -t snapshot -r tank/work
git push -> zfs send -i @before tank/work@nowzfs diff is the one that surprises people. It prints a change type per path in the same shape git uses: + for created, - for removed, M for modified, R for renamed. No repository, no staging area, no prior arrangement. The filesystem simply knows, because it recorded which blocks moved in which transaction group.
Three things follow, and they are the reason I run my machines this way.
Before anything risky, take a snapshot. A system upgrade, a database migration, a configuration change you are not sure about. It costs a moment and no meaningful space, and it converts an irreversible action into a reversible one. zfs rollback puts the dataset back exactly as it was, including the files you forgot were involved. On a machine where the root filesystem is ZFS, bectl does the same for the entire operating system.
Local development gets a safety net underneath the one it already has. Git protects what you committed. It does nothing for the build artefacts, the local database, the container volumes, the half-configured service, the scratch directory you were about to need. A dataset snapshot covers all of it at once, including the state git was never asked to track, and a clone gives you a writable copy of that whole world to experiment in while the original sits untouched.
Between machines, the same snapshots become the transport. Because zfs send -i ships a delta computed from transaction groups rather than discovered by scanning, keeping a second machine in step with the first is a matter of sending the increment since the last common snapshot. The receiving side does not end up approximately in sync. It ends up byte-identical, verified by the same checksums that guard the pool, and it holds the same history you do.
A word on the read-only part, because it is regularly filed as a limitation and belongs in the other column. A ZFS snapshot cannot be modified. Ever. If you want the state to change, you take another snapshot, and what you end up with is a clean linear history of states that actually occurred. This is exactly git's bargain: amend a commit and you do not edit it, you produce a different one with a different hash, because the content is the identity. A snapshot you can quietly rewrite is not a record of anything, and as a safety anchor it is worth precisely nothing, since whatever went wrong on the live filesystem can generally reach it too.
Btrfs concedes the point in its own design. Its snapshots may be writable, and btrfs send requires a read-only one, which is to say the project enforces immutability at exactly the moment a snapshot has to behave like a commit.
I would go further and call the writable snapshot a design weakness, without meaning that as an attack on anybody. Ask what it buys. Whatever you wanted a writable copy for, a clone or a fresh subvolume gives you, and gives you it without overloading the word that everyone uses for their recovery point. So the upside is a shorter command, and the downside is that a restore point can be written to by whatever is running on the machine, which in the interesting cases is the thing you are restoring away from. Ransomware families have spent years learning to hunt down shadow copies and snapshots before they encrypt anything, for the obvious reason. A recovery point that the attacker can edit is not a recovery point, it is a suggestion.
None of which makes Btrfs unsafe, and the fix is one flag: btrfs subvolume snapshot -r behaves exactly as you would want. The objection is narrower and it is about defaults. A guarantee you have to remember to ask for is not a guarantee, it is a habit, and habits are what fail at three in the morning during an incident. ZFS took the option away and left only the correct behaviour.
The counterpart honesty: read-only does not mean indestructible. A ZFS snapshot can still be destroyed by anyone holding the right privileges on the pool, which is why zfs hold exists to pin one against deletion, and why a snapshot only becomes a backup once it has been sent to a machine the first one cannot reach. That is the normal state of affairs in version control rather than a shortcoming: git protects the contents of a commit, since the hash is the identity, and it protects nothing at all against somebody with push rights running gc, rebase or push --force. Immutable means unforgeable in both systems. It has never meant undeletable in either.
Reaching into the past costs about as little. Every ZFS filesystem carries a .zfs directory at its root, hidden by default and controlled by the snapdir property, and under it sits one directory per snapshot containing that entire filesystem as it stood. No checkout, no detached head, no working copy to restore. You change directory into a moment in time and read it with ls, grep, diff and anything else you already know, while the live filesystem carries on underneath you, untouched.
Now the honest boundaries, because an analogy that is not policed becomes a lie.
There is no merge. ZFS can fork a line of history through a clone, and it cannot bring two lines back together; if you need that, you need git, and the two coexist perfectly well. The diffs are per file, not per line, so this replaces nothing about reviewing a change. And a snapshot lives in the pool it was taken in, so it is a working practice rather than a backup until you have sent it somewhere else.
Btrfs does most of this too, and it does it on stable ground: its own status tables mark subvolumes, snapshots, send and receive as OK, and the unstable entry applies to RAID56 alone. What differs is ergonomics rather than reliability. There is no btrfs diff in the command list, so working out what changed between two subvolumes means driving btrfs send --no-data and reading the stream, or reaching for Snapper. There is no btrfs rollback either; the way back runs through subvolume management, setting a default and remounting. Both are entirely doable, and both are the reason a tooling ecosystem grew around Btrfs that ZFS never needed. When one program owns the snapshot, the checksum and the redundancy, the verbs are already there.
What it gives you is a machine whose state is addressable. Not the code, which git already handles. The machine.
The Other Project That Had the Same Idea
Btrfs deserves better than the caricature it usually gets, and getting its history right actually sharpens the point rather than blunting it.
A common belief holds that Btrfs was Linux's answer to ZFS. Chris Mason, who started it in 2007 after moving from SUSE to Oracle, has said otherwise: starting Btrfs was less about ZFS than about making sure Linux could keep up with the enormous storage devices arriving over the following decade. Its technical foundation came from somewhere else again: a paper by Ohad Rodeh of IBM on copy-on-write B-trees, presented that same year. Two projects arrived at the same central idea independently, which is usually a sign that the idea is correct.
And Btrfs has real advantages that ZFS does not. It is in the kernel tree, so there is no module to rebuild and no licence question hanging over it. It is far more relaxed about changing the shape of your storage after the fact: adding a disk, removing a disk, converting between profiles on a live filesystem are all things Btrfs does comfortably and ZFS has only recently and partially learned. For a home server that grows one disk at a time, that flexibility is worth a great deal, and anybody who dismisses it has not run one.
The divergence is at a single point, and it is instructive.
RAID 5 has a failure mode called the write hole. A stripe is written across several disks along with its parity; if the power goes at the wrong instant, some of that stripe is on disk and some is not, and the parity no longer matches the data. Nothing announces this. It is discovered later, usually while rebuilding after a drive failure, when the parity is asked to reconstruct something and reconstructs rubbish with complete confidence.
ZFS does not have this failure mode, and not because it works around it. RAIDZ writes every block as a complete stripe of variable width, sized to the write rather than to a fixed geometry, inside the same transaction that copy-on-write already guarantees. There is no window in which a partial stripe exists, so there is no hole. The OpenZFS documentation states it flatly: RAIDZ eliminates the RAID-5 write hole, in which data and parity become inconsistent after a power loss. That property is not a feature bolted on; it is what you get when the thing writing the parity is the same thing managing the transaction.
Btrfs took the conventional fixed-width stripe layout for its RAID 5 and 6 profiles, and inherited the write hole with it. The consequence is documented by the project itself, and Btrfs is admirably honest about it: in the official status tables, RAID56 is marked unstable, a category the same page defines as do not use for anything other than testing, known severe problems, missing implementation of some core parts. That entry has stood for years. The project began in 2007.
So this is not a story about one team being better than another. It is a story about a decision taken at the foundation, which either makes a class of problem impossible or leaves it for somebody to solve later, and where later has a way of not arriving.
The Resource Argument, Which Is Usually Made Backwards
Every discussion of ZFS eventually produces somebody saying it needs a gigabyte of RAM per terabyte of storage. This is the most durable piece of misinformation in the field, and it comes from a real recommendation that has been detached from its context.
That guidance is about deduplication. The dedup table holds an entry for every unique block in the pool, each costing a little over 320 bytes, and it has to be in memory to be any use. Deduplication is off by default and always has been. Switch it on across a large pool and yes, budget accordingly. Leave it alone, as nearly everyone does, and the rule simply does not apply to you.
The ARC is the other half of the confusion. It is a cache: the Adaptive Replacement Cache, which replaces the least-recently-used algorithm that virtually every other filesystem uses with one that tracks both recency and frequency and adapts between them. It gives memory back when applications want it. What monitoring tools report as consumed is the same memory a Linux box devotes to its page cache, counted under a different name and then held against it.
Meanwhile the actual efficiencies go unmentioned, and they are not small. Compression with lz4 has been on by default since OpenZFS 2.2.0, and compression reduces I/O, because fewer bytes cross the bus than would otherwise. One layer means one set of metadata and one cache rather than three that cannot inform each other. Snapshots need no reserved space. And the replication difference described earlier is not a percentage, it is a category: reading a list of changed blocks against reading the entire dataset to work out what changed.
The Limit
Some honesty, because a piece like this is worthless without it.
RAIDZ is not a backup, and treating it as one is a mistake that ZFS makes especially easy because everything feels so safe. A pool protects you from a disk lying to you. It does not protect you from rm -rf, ransomware, or a fire.
Checksums cost CPU, and on a small box doing heavy sequential work you can measure it. ZFS genuinely likes memory and performs better with more of it, even without deduplication; the gigabyte-per-terabyte rule is wrong, but "give it plenty" is fair advice. Expanding a RAIDZ vdev remains awkward compared with what Btrfs does casually. And if you only ever use Btrfs in its single-device or RAID1 profiles, where the status tables say it is fine, you will have a perfectly good time and none of the above will ever trouble you.
The specific claim here is narrower than "ZFS is better". It is that a system which refused to split itself into two components can answer a question that split systems cannot ask, and that the answer to that question is the difference between silent corruption and a line in an error counter.
Somebody at CERN wrote 2 GB of known patterns to disk and read them back to see what had changed on the way, and the industry has been quietly living with the answer ever since. Most storage stacks handle it by not being in a position to notice.
Twenty years on, the design that stopped drawing a line between two layers is still the one that can tell you which copy was right. That was not luck. It was the consequence of taking one decision seriously enough to build everything else out of it.