Red Hat keeps an article in its knowledge base for a morning nobody plans. Every command on the machine fails with the same line, "relocation error: /usr/lib64/libc.so.6: symbol _dl_starting_up, version GLIBC_PRIVATE not defined in file", ls included, and the article adds what follows a reboot: "Kernel panic - not syncing : Attempted to kill init!" It covers Red Hat Enterprise Linux 6, 7, 8 and 9. Something has put the C library and its loader out of step, and every program on the disk that needs them has stopped, the first one included.
Why one library can stop a machine
The kernel starts exactly one program by itself. On Linux it looks for /sbin/init, which on Debian and Ubuntu is a link to systemd, and when that file names an interpreter in its ELF header, the kernel hands control to the dynamic loader first. The loader finds every library the program asks for, maps it, resolves its symbols, and only then jumps into the program. Should any of that fail, process number one dies before its own code has run, and the kernel has a firm view about a dead process number one: it panics.
Arch Linux users met the same wall in September 2017. An update of glibc from 2.26-1 to 2.26-2 left a machine with "No working init found", reported on 13 September and cured by 2.26-4 the next morning. Four months earlier the rescue itself had gone the same way on 32-bit machines: the busybox inside Arch's initramfs, the very shell meant for a boot that has gone wrong, crashed after a glibc update, and the maintainer's remedy was to link it statically.
November 2003
FreeBSD had this conversation twenty-three years ago this week. On 19 November 2003 Gordon Tetlow committed a one-line change to the Makefile of init, with the message "Make init statically linked by default. It's not worth the pain of having a dynamically linked init as recently seen by ia64 woes." The release engineers signed it off, and the line in releng/15.0 still reads NO_SHARED?= YES.
The timing was deliberate. FreeBSD 5.2, released on 12 January 2004, was the release in which "many executables in /bin and /sbin are now built using dynamic, rather than static linking", and its notes give the reasons: loadable PAM and NSS modules for the base utilities, and a smaller root file system. The project wanted shared libraries in the root and took them, with two exceptions. One was init. The other was a new directory, /rescue, written by Tim Kientzle, which holds statically linked copies of the tools needed to repair a system and is rebuilt with every buildworld. Its manual page names the worry without ceremony: "there is a real possibility that the standard tools in /bin and /sbin may become non-functional due to a failed upgrade or a disk error."
I unpacked the base.txz of FreeBSD 15.0 this morning to see what that decision looks like today. sbin/init is a statically linked executable of 1,245,488 bytes. /rescue holds 150 entries, and 149 of them are hard links to a single statically linked file of 19,939,888 bytes, stitched together by crunchgen; the odd one out is dhclient-script, which is a shell script and needs no linking at all. Among the 149 are sh, vi, mount, fsck_ffs, tar and fetch, and with them zfs, zpool, zdb and bectl. A machine whose libraries a bad upgrade has wrecked can remount its root with /rescue/mount -uw / and then hand the next boot to the boot environment it had before the upgrade with /rescue/bectl activate, without loading a single shared object. Belt and braces, and FreeBSD has worn both since 2004.
What debian's first process loads
The same morning I unpacked systemd 257.13 from Debian 13 and read its ELF headers with a small parser of my own. /usr/lib/systemd/systemd names /lib64/ld-linux-x86-64.so.2 as its interpreter and five libraries as its needs, two of which are systemd's own libsystemd-core-257.so and libsystemd-shared-257.so, and those bring needs of their own. Count one level further down and the first process of a Debian machine turns out to need fifteen distinct libraries from twelve projects before it can start: libc and libm, libsystemd-core and libsystemd-shared, libselinux and libapparmor, libaudit, libseccomp, libcap, libacl, libmount and libblkid, libcrypt, libpam, and libcrypto from OpenSSL. Each arrives as a package with its own upstream and its own calendar, and any one of them can now stop the boot.
A security fix in any of those libraries reaches process number one with the next package upgrade and without a rebuild of systemd, which is the whole case for shared libraries. The systemd developers have seen the other side of it too. Since version 262 of 22 September, systemd "can now be built as a single statically linked PID 1/executor binary suitable for very small containers", and the note adds that such builds "do not use dlopen() to load optional libraries at runtime and use simplified passwd/group file lookup instead of NSS". It is a build option, offered for containers, and a Debian server still installs the dynamic binary.
On Linux the answer to a broken root is the initramfs, a small image the boot loader brings along, carrying its own copies of the libraries and a shell. It works, touch wood, and it is a capable tool. It is also a second little system, rebuilt by a hook every time the kernel or a relevant package changes, and on Debian 13 the full busybox inside it is only a recommendation of initramfs-tools, with the smaller klibc tools as the floor. Debian does ship a statically linked rescue shell, busybox-static, "intended to be used as a rescue shell, in the event that you screw up your system", as an optional package that someone has to have installed before the morning in question.
The limit
Static tools do less. rescue(8) says so plainly: the tools in /rescue "do not have full use of the locale, pam(3), and nsswitch libraries". They exist for repair, and nobody logs in through them.
A fix in libc reaches init through a rebuild. On FreeBSD that rebuild is the ordinary update of the base system: init and /rescue come from the same tree as libc and are replaced together with it, by freebsd-update or by pkg, so the static copies and the shared library never drift apart.
The failure is rare. A careful administrator with snapshots of the root may never need either remedy. What the system leaves on the disk decides how that one morning goes.
The receipt
Today's default on a Linux server starts the first process through the dynamic loader and fifteen libraries from twelve projects, and keeps its remedy in a separate image that a hook rebuilds, plus an optional package somebody may have installed. The receipt goes to the administrator who meets the relocation error: a live medium, a chroot, the right version of glibc and a rebuilt initramfs, roughly in that order.
On FreeBSD the receipt is two lines typed from /rescue, mount -uw / and bectl activate with the name of yesterday's boot environment, followed by a reboot.
The kernel will not let the first process die, so FreeBSD saw to it that the first process needs nobody else to live.