The Unix Way ■ Episode 14
Between the firmware that knows almost nothing and the kernel that must know everything, there is a small program with a rather strange job. The bootloader is the one piece of software whose design quietly encodes what the operating system above believes itself to be.
Three answers to the same question. Each tells a different story.
FreeBSD's Loader
Three stages.
boot0
is a 446-byte master boot record. boot1 reads
the partition table and locates loader.
loader(8)
is a Forth interpreter, around 600 KB on amd64.
Forth, because in 1992 someone rather sensibly decided the bootloader should be programmable without itself becoming an operating system. Forth gives you a small, deterministic, stack-based language with no runtime assumptions about memory layout or operating system services. It is enough to express "look at this dataset, decide which kernel to load, present a menu, hand off", and not enough to tempt anybody into reinventing libc above it.
The loader understands UFS and ZFS natively. It can mount a ZFS dataset as root. It knows what a Boot Environment is. It presents the list of available Boot Environments as a menu before the kernel starts. None of this is bolted on. It is the design.
The pre-install bootloader for ZFS-on-root systems writes the loader into a small partition outside the ZFS pool, but the loader itself reads the pool metadata, walks the dataset hierarchy, and offers the user the choice of which root dataset to boot. This is the enabling primitive on which the entire FreeBSD Boot Environment workflow stands.
What the Userland Built
The loader exposed Boot Environments as a first-class concept. The userland question was: how does an admin create, list, activate, destroy, and switch between them?
The first answer was a shell script called manageBE,
written in the early days of ZFS-on-FreeBSD. Functional,
but not pleasant to use.
In 2012, a Polish FreeBSD admin called
vermaden
wrote
beadm(8)
in POSIX sh(1) and awk(1),
deliberately mimicking the Solaris and Illumos
beadm interface so that anyone coming from
those systems would feel at home. The original announcement
and discussion lives on the FreeBSD Forums in the thread
"HOWTO ZFS Madness"
(number 31662, still readable today). The script was, and
remains, around fifteen hundred lines of POSIX shell and
awk: small, auditable, dependency-free, runnable on any
FreeBSD system since 9.x.
For six years, vermaden's beadm was the working
standard for managing Boot Environments on FreeBSD. It was
packaged in ports, recommended by the documentation, and
built into countless admin workflows.
In 2018,
FreeBSD 12.0 shipped bectl(8),
a reimplementation in C that landed in base. The motivation
was straightforward: a tool used as universally as
beadm had become deserved to be in base, with
all the testing and consistency guarantees that base
implies. The C rewrite gave it tighter integration with
libbe, the FreeBSD library that already
encoded the Boot Environment data model, and direct ZFS
access without the cost of subprocess fan-out.
The transition was not entirely smooth. Early
bectl versions had bugs that beadm
did not, in part because beadm had been shaken
out by six years of production use across hundreds of
admins. There are documented cases from the 12.x era in
which running beadm against a
bectl-confused dataset state cleanly resolved
the inconsistency. The bugs have largely been fixed, and
bectl is now the recommended tool.
beadm continues to be developed and continues
to ship features that bectl has not adopted.
The most interesting is the
REROOT option:
after creating a Boot Environment, upgrading the running
system, and discovering that something is not right,
REROOT swaps userspace into the pre-upgrade
Boot Environment without a full reboot. The kernel
stays up; the userspace is reloaded from the chosen
dataset. The whole operation takes seconds rather than the
half-minute of a reboot. It is the kind of feature that
exists because someone was solving an actual operational
problem and writing the code that fixed it, rather than
waiting for the perfect abstraction.
That trajectory, from manageBE to
beadm to bectl, is a model of how
Unix tools mature. A clever script meets an unfilled need.
A wider community adopts it and shapes it. A reimplementation
in base preserves what worked and fixes what did not, while
the original keeps innovating in places where base cannot
move as quickly. Both tools coexist, and the choice between
them is technical rather than political.
Linux: LILO (1992 to 2015)
Werner Almesberger wrote LILO at ETH Zürich in 1992 and maintained it until 1998. John Coffman took over until 2007, when development passed to Joachim Wiedorn. Active development ended in December 2015 with version 24.2, and was never resumed.
The Linux Loader's design choice was a blocklist.
The position of the kernel image on disk was recorded as a
list of physical sectors at install time. Move the kernel,
the bootloader points at gibberish. Update the kernel, run
/sbin/lilo before rebooting or the machine
refuses to come back. One does, in retrospect, find this a
trifle hopeful.
This was reasonable in 1992. Disks were small, kernels were rebuilt rarely, and the BIOS understood little beyond INT 13h. The assumption was a slow, static world.
By 2010 the assumption had become a trap. GPT and UEFI made blocklists structurally awkward. RAID and LVM moved blocks behind the back of the bootloader. The Linux kernel started shipping new versions every couple of months. The ritual that LILO required after each change had become the leading cause of unbootable systems for users who forgot it. The world had moved on; the bootloader had not.
Linux: GRUB (2005 to present)
The other extreme. GRUB began in 1995 as a research bootloader by Erich Boleyn, was adopted by the GNU project in 1999, and was rewritten from scratch as GRUB 2 in 2005. The result is a small operating system that runs before the operating system.
GRUB 2 ships filesystem drivers for ext2, ext3, ext4, btrfs, XFS, F2FS, JFS, ReiserFS, FAT, NTFS, ISO9660, AFFS, HFS, UDF, ZFS (read-only, partial), and several others. It can decompress kernels in zlib, lzma, lz4, and zstd. It runs scripts in its own POSIX-shell-flavoured language. It supports themes, fonts, gfxmode, network boot, UEFI Secure Boot, multiboot, and chainloading. Its core is around 200 KB; its loadable modules add several megabytes.
Cater a holiday camp where every child has a different diet and you will need a vast kitchen, a vast larder, and rather a lot of cooks; the food will feed everyone competently and excel at none. Linux distributions disagree about filesystem choice, kernel layout, root-on-everything, and boot configuration. GRUB has to understand every variant because the world above it does not converge.
The cost of that assumption is everything complexity costs. The benefit is that GRUB will boot almost anything you put in front of it.
The omission is interesting. GRUB has read-only ZFS support, sufficient to load a kernel from a ZFS dataset. It does not have Boot Environment awareness, because the conversation about ZFS-as-an-OS-management-layer never quite happened on Linux. ZFSBootMenu exists precisely to fill that gap, by replacing GRUB entirely with a small kexec-loaded Linux kernel whose only job is to present a Boot Environment menu and hand off to the chosen one. Rather circular, that.
The Point
The question is not which bootloader is best. It is what the bootloader assumes about the system above it.
FreeBSD's loader assumes the system above it is coherent
enough to be worth talking to: that the bootloader, the
filesystem, the kernel, and the userland are designed by
people who can sit in the same room. From that assumption,
things like Boot Environment selection at the loader become
possible, and tools like beadm and
bectl become natural.
LILO assumed a slow, static world in which the admin would re-run the installer after each change. The world was no longer that world by 2005, and the bootloader could not move with it.
GRUB assumes a fragmented world in which the bootloader must understand every variant of every filesystem because the distributions above it do not converge. The cost is permanent complexity, the benefit is that GRUB boots anything.
Three loaders. Three theories of the operating system. Each design is correct for the world it assumes. The interesting work is choosing which world to live in.
FreeBSD loader: ~600 KB Forth, UFS+ZFS native, Boot Environment menu before kernel start.
beadm(vermaden, 2012, POSIX sh+awk) →bectl(FreeBSD 12, 2018, C in base); both coexist, choice is technical. LILO (1992-2015): block lists, re-run /sbin/lilo or the machine declines to boot. GRUB 2 (2005-): 15+ filesystem drivers, scripting, themes, catering for a fragmented world. ZFSBootMenu fills the BE gap by replacing GRUB entirely.