Vivian Voss

ZFS Rebuilds Only What You Wrote

like for like freebsd zfs linux

The courier brought the replacement disk at half past ten, which was decent going for a fault reported at three in the morning. The server it was meant for had spent the night on one leg: two 16 TB disks in a Linux mirror, one of them now a paperweight, and about two terabytes of actual data between them.

Out comes the dead disk, in goes the new one, and a single command puts it to work:

mdadm --manage /dev/md0 --add /dev/sdb1

Then /proc/mdstat shows a progress bar, and the progress bar is in no hurry. md copies the surviving disk onto the new one from the first sector to the last, all sixteen terabytes of it, and the kernel's default ceiling for that copy is 200,000 KiB a second. At that pace the job takes the best part of a day (a little under twenty-two hours, provided the disks keep up), and for every one of those hours the two terabytes live on a single disk.

What the mirror knows

md is good at its job, and its job is narrow on purpose. It sits underneath the filesystem and presents two disks as one block device, so ext4 or XFS on top never learns there are two. That is why mdadm has lasted as long as it has. It works under any filesystem and any encryption layer, and it asks nothing of whatever lives above it.

The price of that modesty is ignorance. md knows sectors, and it cannot tell which of them hold a file and which hold nothing at all, so a new disk has to receive every one. mdadm offers a remedy and recommends it whenever an array is created: "To optimize recovery speed, it is recommended to enable write-intent bitmap". The bitmap remembers which regions were written while a disk was away, so a disk that drops out for an hour and comes back needs only that hour's changes. For a brand-new disk it is no help whatsoever, and the kernel's own source says so in as many words: "By default, the old bitmap framework will recover all data".

There is a second thing md does not know. It keeps no checksums, so it copies whatever the surviving disk hands over and has no means of telling a good sector from a quietly rotten one. On the morning in question, the one disk left standing is the only authority there is.

Who knows what is on the disk Debian 13, mdadm ext4 or XFS knows the files md RAID1 knows sectors, no checksums disk new disk new disk receives every sector, empty or not, unchecked FreeBSD 15.1, ZFS ZFS filesystem and volume manager in one layer: knows the live blocks and the checksum of each disk new disk new disk receives the live blocks, each checked on the way The layer that copies has to know what is in use, or it copies everything

The same disk on FreeBSD

On FreeBSD with a ZFS mirror the morning also takes one command, and since the new disk sits in the bay of the old one, it needs only the old name:

zpool replace tank ada1

What follows is different, because ZFS is the volume manager and the filesystem in one piece of software. It knows which blocks belong to a live file, since it keeps the tree that points to them, and the resilver walks that tree from the top. Space that holds nothing is never visited. Every block it does copy is read together with its checksum and checked before it reaches the new disk, and when the surviving copy fails the check, ZFS reports the bad block and keeps it off the new disk.

I tried it this morning, 17 November 2026, on a FreeBSD 15.1 machine with OpenZFS 2.4.2. Two files of 4 GiB each made a mirrored test pool. I wrote 600 MiB of random data into it, deleted half of that again, and replaced one side of the mirror with a fresh, empty file:

scan: resilvered 301M in 00:00:01 with 0 errors

Afterwards the new file occupied 307,861 KiB of real space, against 615,569 KiB on the side it replaced. The 300 MiB I had deleted stayed behind on the old disk, and the 3.4 GiB of empty space was never touched. md, given the same pair, would have copied all 4 GiB.

What a new disk receives in a 4 GiB mirror md, default bitmap 4,096 MiB Ever written 600 MiB, deleted half included ZFS resilvered 301 MiB, every block checked, 0 errors md by design, from the kernel source: "By default, the old bitmap framework will recover all data" Own measurement, 17 November 2026, FreeBSD 15.1, OpenZFS 2.4.2

Scale that back up to the courier's morning and the arithmetic is no great shakes: the copy shrinks to the two terabytes that hold data, and every byte of them is verified. Two terabytes is still a fair old while, but at least it is the right two.

The rest of the job lives in the same tree. The FreeBSD installer offers a ZFS mirror for the root disk. zfsd, part of the base system since FreeBSD 11.0, listens for a failed disk and brings in a hot spare, and with the pool property autoreplace=on it replaces a disk slotted into the same bay without being asked. periodic mails the health of every pool each night once daily_status_zfs_enable="YES" stands in periodic.conf. None of these comes from somewhere else, and each of them knows the others are there.

The new bitmap

In the autumn of 2025 Linux 6.18 took a first step towards the same idea and gave md a second kind of bitmap, the lockless llbitmap written by Yu Kuai. Its description in the kernel source names exactly what a replaced disk needs: "support only resync or recover written data, means in the case creating new array or replacing with a new disk, there is no need to do a full disk resync/recovery". mdadm 4.6 learnt to create such arrays in March 2026 with --bitmap=lockless, which its manual describes as "experimental".

It is a genuine step, and it shows what the old habit cost. It also shows what md still cannot know. The new bitmap records where data has ever been written, a faithful history of the disk, and a deletion in the filesystem reaches it only if a discard travels down through every layer first. In my test the deleted 300 MiB would most likely have made the journey too. It checks no checksums either. And the ordinary Debian 13 server runs kernel 6.12 with mdadm 4.4, which have neither, while a current mdadm that asks its usual question still sets up the old bitmap.

Linux does have filesystems that know their own blocks. Btrfs replace will "duplicate the data to the target device which is currently stored on the source device", and OpenZFS runs on Linux as a module outside the kernel tree, compiled again for every kernel update. Both prove the design right. On FreeBSD that design has been in the base system since FreeBSD 7.0 in 2008, offered by the installer, with its fault daemon beside it.

Knowing what to copy, a short history 2008 ZFS in FreeBSD 7.0 2016 zfsd in FreeBSD 11.0 2025 llbitmap in Linux 6.18 March 2026 mdadm 4.6, "experimental" Nov 2026 Debian 13: kernel 6.12, mdadm 4.4 Green: FreeBSD base system · Accent: Linux md

The difference

Like for Like            Debian 13, mdadm 4.4        FreeBSD 15.1, OpenZFS 2.4.2
Command                  mdadm --add                 zpool replace
What is copied           every sector of the disk    the blocks in use
Checked on the way       nothing                     every block against its checksum
Own test, 4 GiB mirror   4 GiB (by design)           301 MiB, 0 errors
Deleted data             copied with the rest        left behind

Both sides type one command, and only one of them gets back a copy of the data, checked by the software that wrote it.

The limit

A nearly full pool narrows the gap. When the data fills most of the disk, ZFS has nearly as much to copy as md, and a fragmented pool reads in the order of its block tree, which is not the order of the platter. OpenZFS has put its reads in order before issuing them since version 0.8, and on a mirror zpool replace -s copies sequentially, then starts a scrub, because "Checksums are not verified during sequential reconstruction".

RAID-Z has no sequential option. The manual says it plainly: "Sequential reconstruction is not supported for raidz configurations." A RAID-Z resilver always walks the tree.

md fits under anything. A stack that must be LUKS with XFS on top is md's home ground. ZFS wants to be the whole stack, and on FreeBSD it brings its own encryption and its own volumes along to make that a fair exchange.

Linux can get there with Btrfs or OpenZFS. Then the administrator keeps that choice running by hand, a module beside every kernel or a filesystem beside the distribution's default. On FreeBSD it arrives with the release.

A mirror that does not know what it holds has to copy everything it might be holding.