On the fourteenth of September, a little after eight in the morning, the machine that serves this blog went from FreeBSD 15.0 to 15.1. This morning I asked it what it remembered of the occasion.
% bectl list
BE Active Mountpoint Space Created
15.0-RELEASE-p13_2026-09-14_080622 - - 1.43M 2026-09-14 08:06
15.0-RELEASE_2026-09-11_085654 - - 140M 2026-09-11 08:56
15.0-p13-vor-15.1-20260914-0800 - - 8K 2026-09-14 08:00
15.1-RELEASE-p3_2026-09-14_081056 - - 1.03M 2026-09-14 08:10
15.1-RELEASE-p3_2026-09-14_081212 - - 4.15M 2026-09-14 08:12
default NR / 1.75G 2026-09-07 19:40Six lines, and five of them are ways back. Four were laid by freebsd-update on its own: one before a patch on the eleventh, and three on the morning of the upgrade, one for each pass of the install. The fifth is mine, taken by hand at eight o'clock sharp and named in German, which says something about the state of mind at the time. Each of the five is the complete system as it stood at that minute, kernel and base system and packages and configuration, and all five together occupy about 147 megabytes beside a system of 1.75 gigabytes.
Nothing had to be installed to get them, and nobody had asked for four of them.
The way back on a Linux server
A Debian 13 server, installed with the defaults, leaves something smaller after the same morning. The installer chooses ext4, and apt keeps two kernels, which its manual describes as the running one and the latest. If an update breaks the machine, the boot menu offers the previous kernel under "Advanced options", and that is the way back: one component out of several hundred, and usually the one that did not break. The C library, OpenSSL, systemd and whatever configuration the package scripts rewrote stay precisely where the update left them. Ubuntu's installer chooses ext4 as well. Red Hat Enterprise Linux keeps three kernels, because installonly_limit defaults to 3, and dnf history undo reverts the package changes of a transaction. Beyond that lies the backup.
There is a sound reason for all of this. A way back for the whole system needs a filesystem under the root that keeps the old blocks while it writes the new ones, and ext4 and XFS were never built for that. ZFS is kept out of the Linux kernel by its licence. Btrfs can do it, and Red Hat deprecated it in RHEL 7.4 and removed it in RHEL 8, with the release notes adding, in passing, that the snapper package went with it. Fedora Workstation has installed btrfs by default since release 33, and its change proposal said so plainly at the time: "There will be no automatic snapshots/rollbacks." Ubuntu built zsys to manage boot environments on ZFS, and in April 2022 Canonical removed it from the installer for the LTS, noting that it was "in very low maintenance mode (critical bug fixes only)".
The other side has a strong case, and it should be heard in full. openSUSE has shipped btrfs with snapper as its default since 13.2 in November 2014: zypper takes a snapshot before and after every transaction, and GRUB will start a read-only snapshot on request. NixOS adds an entry to its boot menu every time its configuration changes and, with GRUB, keeps up to a hundred of them. Fedora's atomic editions and Red Hat's image mode keep the previous system image and return to it with one command. The people who built these understood the problem perfectly well, and each of them built the answer for one distribution, where it has stayed.
What FreeBSD lays down first
The installer's first partitioning choice, "Auto (ZFS)", lays the root out for boot environments from the start. Before freebsd-update installs anything, it reads one line of its configuration file: CreateBootEnv yes, shipped as the default since 12.3 and 13.1. With that line in place it calls bectl, which snapshots the root dataset and clones it under a name made of the running version and the time of day, and that is where the names in the list above come from. It stands aside in the cases where it could not help: when the root is not on ZFS, when the layout is not arranged for boot environments, when it runs inside a jail, and when it is patching one from outside.
A clone costs nothing on the day it is made. It shares every block with the system it came from and grows only by what changes afterwards, which is why the entries from the upgrade morning cost between eight kilobytes and four megabytes each, and why the oldest, which predates the whole point release, now holds 140 megabytes that the running system no longer shares.
The loader reads ZFS without any help from an operating system. When the pool holds more than one boot environment, its menu offers an entry called "Boot Environments", and the choice is made before any kernel has started, so a system that no longer comes up can still be sent back. The same menu carries a second entry for the work a boot environment does not cover: after a zpool checkpoint, it offers to rewind the entire pool to the moment the checkpoint was taken.
The tool has a pedigree. vermaden wrote beadm in 2012, in POSIX shell, after the Solaris command of the same name. In 2017 Kyle Kneitinger turned the idea into libbe and bectl as a Google Summer of Code project, with Allan Jude as his mentor, and bectl has been part of the base system since FreeBSD 12.0. Its manual records, with some courtesy, that it was derived from beadm.
Four things a boot environment will do
Rolling back is the part everybody demonstrates. What tends to go unmentioned is everything that can happen before anybody reboots.
The first is bectl jail, which starts any boot environment as a jail, with a shell inside it, whilst the running system carries on serving. On the default layout the packages and their database live inside the boot environment, so next week's system can be upgraded and inspected before it has been booted once:
# bectl create next
# bectl jail -o ip4=inherit -o ip6=inherit next pkg upgrade
# bectl activate -t next
# shutdown -r nowThe second sits in the third line. The -t makes the new environment the choice for the next start and for no other. If it comes up as it should, bectl activate next makes the arrangement permanent. If it hangs, the power switch is the rollback, because the following start returns to the old system by itself and nobody has to reach a menu at all.
The third concerns the machine next door. bectl export writes a boot environment to standard output as a ZFS stream and bectl import reads one in, so a system that has been patched and tested on one host can travel to another through a pipe and arrive identical, block for block:
# bectl export next | ssh standby bectl import nextThe fourth needs a port, and it is the one vermaden drew my attention to. beadm reroot activates a boot environment and then calls reboot -r, which kills every process, unmounts the filesystems, mounts the new root and begins the startup sequence again, all under the kernel that is already running. The hardware is never reset. The source code marks the moment with a comment I would not dream of improving: no messages are displayed at that point, "because Kansas is going bye bye".
Who profits, who pays
The bill for a missing way back is paid in hours, and it tends to fall due at night.
The savings on the other side are real, and every one of them is a sensible decision taken by somebody responsible for one part. The Linux kernel does not have to carry ZFS. Red Hat no longer maintains btrfs, nor the snapshot tool that left with it. Apt never needs to know what filesystem it is writing to, and Canonical stopped carrying a tool it had already put into very low maintenance mode. Each project kept its own share of the work small, and the sum of those small decisions is a server on which the way back is something the operator builds.
The operator can pay in advance. That means installing and minding another project, ZFSBootMenu with zectl or a hand-assembled snapper, on its own schedule and with one more seam between parts that were never designed together. Or it means Red Hat's image mode, where package-mode systems, in Red Hat's own words, "can not be converted to image mode", so the price is a reinstall and an image pipeline to feed it. Even then the way back keeps office hours: after a rollback, according to the documentation, the update timer runs within one to three hours and puts the system back on the image it was rolled back from.
The operator who paid nothing in advance pays at three in the morning, after a library update, in front of a boot menu that offers the one component that did not break.
FreeBSD paid for the same capability once, in one tree. The loader that reads ZFS, the updater that calls bectl before it writes, the jail that can host a system before it boots and the filesystem beneath all of them come from one project and are released and tested together. That is the argument of my book, Integrated by Design, in one concrete case.
The limit
Four limits, none of them small print.
A boot environment needs its root on ZFS. An installation on UFS keeps a backup kernel and the freebsd-update rollback command, which removes the most recent updates, and that is still a modest step beyond two kernels in a GRUB menu.
Data outside the boot environment stays where it is, by design: home directories, logs, mail spools and a database on a dataset of its own. A schema that the new version migrated stays migrated after a rollback. openSUSE and Red Hat's image mode draw the same line around /var.
Some Linux systems do roll back the whole system. openSUSE has done so by default since 2014, NixOS does it with every rebuild, Fedora's atomic editions keep the previous image, and so does RHEL's image mode. Everything above concerns the two large families, Debian with Ubuntu and Red Hat, as their installers set them up.
A reroot does not replace the kernel. Like systemd's soft-reboot, which arrived in version 254 in 2023, it restarts the userland under the kernel that is already running, and a new kernel still wants a real restart.
The fifth way back on that machine is still the one named in German, taken by hand at eight o'clock sharp for eight kilobytes. By twelve minutes past, the updater had laid three more of its own. On FreeBSD the way back is laid by the same program that takes you forward.
On FreeBSD, freebsd-update clones the whole system before it changes a file, and the machine that serves this blog keeps five such ways back for 147 MB. bectl can upgrade the next system as a jail and try it once, with the power switch as the way back. Debian keeps two kernels and RHEL three; the Linux systems that do roll back whole systems each built the answer for themselves.