Every migration plan has a spreadsheet, and every spreadsheet has the row nobody wants to discuss. The web server has a FreeBSD package, so has the database, and so, in a ports tree of 38,870 entries, has very nearly everything else a server is ever asked to do. What is left over is, more often than not, nobody's idea of a household name: the vendor's licence daemon or the agent the auditors insist on, delivered once a quarter as a tarball with linux-amd64 in its name and a support statement listing precisely which distribution releases and kernel versions it is prepared to tolerate. Upgrade the distribution, or merely the kernel, and on Linux it is quite often this very program that falls over first, which makes it a curious thing to stay loyal to. On most teams that row is nevertheless where the plan quietly ends.
The tarball, as it happens, runs on FreeBSD, and the ports tree does this as a matter of routine: the OpenTelemetry Collector is packaged from its Linux build, and the Folding@home client is a Linux binary that starts at boot from an rc script like any other FreeBSD daemon.
A game of Doom, 1995
The copyright lines are still in the source. Thirteen files under sys/compat/linux in FreeBSD 15.1 carry the line Copyright (c) 1994-1995 Søren Schmidt or a near relation of it, and the FreeBSD Foundation, writing about the layer in 2021, gave the motive without embarrassment: "the initial motivation for Linuxulator was to play the game Doom on FreeBSD. From this noble beginning in FreeBSD 2.1 (1995 for those keeping score at home)…" The 64-bit binaries followed in FreeBSD 10.3 in 2016, and arm64 in 12.0.
The mechanism has not changed its shape since. A Linux program is an ELF file like a FreeBSD one, marked with a different brand. When the kernel loads it, the process receives a Linux system call table of its own, and each call is translated into the FreeBSD call that does the same work. The processor runs the program's own instructions. There is no second kernel and no emulated CPU, which is what linux(4) means when it promises to run "many unmodified Linux applications without the need for virtualization or emulation". Paths are looked up under /compat/linux first, so the program finds its own libraries there and the rest of the system where it expects it. Some of the translations are pleasingly direct: a Linux server waiting on epoll is, underneath, waiting on kqueue, because linux_event.c turns every epoll registration into a kevent.
To a Linux binary the machine introduces itself as Linux 5.15.0, the version compiled into linux_mib.h, which is about as much pretending as the arrangement requires.
One target that holds still
The libraries the program expects come as one package. pkg install linux_base-rl9 places a Rocky Linux 9 userland, version 9.8 in the current package set, under /compat/linux, and it is the same package on every supported FreeBSD release. A vendor who builds for Red Hat Enterprise Linux 9 or one of its clones has, without meaning to, built against the libraries FreeBSD provides as well.
FreeBSD keeps its own binaries just as still. The standard kernel of 15.1 carries COMPAT_FREEBSD4 through COMPAT_FREEBSD14, so programs built for FreeBSD 4.0 in March 2000 still find the system calls they were compiled against. The ports tree builds each branch's packages on its oldest supported release, 14.4 for the 14 series and 15.1 for 15, and the rule says why: "to ensure ABI and KBI backwards compatibility within each major branch". The oldest system anyone builds on is a release that still receives security patches.
The Linux kernel keeps a strict rule of its own: its maintainers refuse, loudly and on principle, to change a system call in a way that breaks a running program, and a static binary from many years ago will usually still start on a current kernel. The trouble lives one floor up, in the userland that every distribution assembles for itself. The GNU C library versions its symbols, and since release 2.34 in August 2021 every dynamically linked program asks for the start-up routine in its 2.34 version, so a binary built on a current distribution stops on an older one with version 'GLIBC_2.34' not found. Anyone who wants one binary to reach every distribution therefore goes through the palaver of building on the oldest one still in circulation. Python's packaging rules for Linux say so in as many words: "distributing pre-compiled binaries on Linux is challenging because of the diversity of Linux-based platforms", and wheels "have to be built on carefully-chosen platforms of the oldest possible vintage". The most widely compatible of those platforms, manylinux2014, is CentOS 7 with a C library released in December 2012, and its maintainers note that "CentOS 7 reached End of Life (EOL) on June 30th, 2024". A second set of build images exists for distributions built on musl, the other C library, because a binary linked against glibc will not start there.
Linus Torvalds described the situation at DebConf in 2014, speaking about the dive log program he maintains: "You don't make binaries for Linux, you make binaries for Fedora 19, Fedora 20", and a little later, "all the distributions come in and they screw it all up because they break binary compatibility left and right". The Linux Standard Base was meant to settle exactly this. Its last version, 5.0, appeared in June 2015, and that summer Debian took its leave of it, noting that "There are only _8_ applications by 6 companies on the LSB certified applications list". By September its maintainer called the change "the 'outright giving up' of LSB support for Debian".
The service jail
Here is the arrangement for a Linux-only service, step by step, on a host that already keeps a jail template on ZFS as the Handbook describes. The host loads the layer once:
# sysrc linux_enable="YES"
# service linux startThe service gets a thin jail, a ZFS clone of the template that costs no disk space until something is written, and the Linux userland goes into it from the host, because the jail itself will never be able to fetch anything:
# zfs clone zroot/jails/templates/15.1-RELEASE@base zroot/jails/containers/svc
# pkg --rootdir /usr/local/jails/containers/svc install -y linux_base-rl9
# tar -xf vendor-svc-linux-amd64.tar.gz -C /usr/local/jails/containers/svc/opt
# pw -R /usr/local/jails/containers/svc useradd svc -u 1001 -d /nonexistent -s /usr/sbin/nologin
# mkdir -p /var/run/svc /usr/local/jails/containers/svc/var/run/svc
# chown 1001 /var/run/svcThen the jail itself, with example names:
svc {
path = "/usr/local/jails/containers/svc";
host.hostname = "svc";
ip4 = disable;
ip6 = disable;
mount.devfs;
mount += "linprocfs $path/compat/linux/proc linprocfs rw 0 0";
mount += "linsysfs $path/compat/linux/sys linsysfs rw 0 0";
mount += "tmpfs $path/compat/linux/dev/shm tmpfs rw,mode=1777 0 0";
mount += "/var/run/svc $path/var/run/svc nullfs rw 0 0";
exec.jail_user = "svc";
exec.start = "/opt/svc/bin/svcd --listen /var/run/svc/svcd.sock &";
}Each line has a job (the hostname's job is mostly to look tidy in jls). ip4 = disable and ip6 = disable stop the jail from using either protocol at all, in the words of jail(8), so the service has no address, no route and nobody to telephone. The only way in or out is the socket directory the host mounts with nullfs, and the host decides who else may see it. The three Linux filesystems give the program the /proc, /sys and shared memory it expects, mounted by the host before the jail starts. The daemon starts without a shell: jail(8) hands exec.start to /bin/sh only when the line contains a character a shell would have to interpret, and a path, two arguments and a trailing ampersand contain none, so the binary is executed directly. Inside, root keeps 60 of the 244 privileges FreeBSD 15.1 defines, and the service does not even run as root. Resource limits are one more line each, once kern.racct.enable=1 stands in loader.conf:
# rctl -a jail:svc:memoryuse:deny=2g
# rctl -a jail:svc:maxproc:deny=64Patching happens from outside. pkg --rootdir upgrades the Linux userland with the same command that installed it, and the FreeBSD base underneath is updated with freebsd-update -j or re-cloned from a patched template, the two ways the Handbook offers for thin jails.
What the service cannot reach
The Linux table in FreeBSD 15.1 has 365 entries. I counted them against the source this morning: 270 are implemented, fifteen are obsolete entries that a current 64-bit Linux program does not call, and 80 are placeholders that answer every call with ENOSYS, the error for a system call that does not exist.
Among those 80 are the interfaces that the Linux world itself has spent the last few years fencing in.
io_uring is one. In June 2023 Google reported that "60% of the submissions exploited the io_uring component of the Linux kernel" in the past year of its kernel bounty, around a million dollars out of 1.8 million paid, and added: "It is disabled on production Google servers." The Linux kernel documentation now carries a switch to refuse it, kernel.io_uring_disabled, explained in one sentence: "Enabling this shrinks the kernel's attack surface." By default it stays at 0, which leaves io_uring open to every process. Unprivileged user namespaces, reached through unshare and setns, are another. Ubuntu has restricted them by default since 24.04, after Canonical wrote that they "are now broadly used as a step in several privilege escalation exploit chains". The bpf call is a third, closed to unprivileged users by default since Linux 5.16 because of speculative-execution side channels. And userfaultfd, limited for unprivileged users since Linux 5.11, was described in a kernel commit of 2019 as a way "to stall a kernel thread", so that a race in an exploit stays open for as long as the attacker needs. Module loading and kexec are on the list as well.
The layer answers all of them with ENOSYS, and that is where they belong. Implementing them would import precisely the interfaces that Linux's own distributors, and Google on its own servers, have spent the last few years switching off or fencing in.
The strongest objection
A Linux administrator will reply that none of this needs FreeBSD, and the reply deserves to be heard in full. Docker's default seccomp profile already refuses io_uring, userfaultfd, keyctl and kexec, and it allows bpf, unshare, setns and module loading only with capabilities that a default container does not have. systemd offers SystemCallFilter=, RestrictNamespaces=, PrivateNetwork= and ProtectSystem= for any service that wants them. Both are real protection.
Both are also filters, and that is the difference that remains. The kernel code is still there underneath, and the filter that keeps a program away from it is written into a profile or a unit file by whoever deploys the service. --privileged lifts it, seccomp=unconfined lifts it, and a systemd unit without a SystemCallFilter= line never had one. FreeBSD's Linux layer has no such code underneath, so there is nothing to lift.
The limit
It is not a whole Linux. No systemd, no containers inside the jail, no Linux kernel modules, and a program that insists on one of the 80 missing calls will say so the first time it starts. The layer is built to run a service, and that is how this piece uses it.
The layer has bugs of its own. Nine FreeBSD security advisories since 2000 carry linux in their title, two of them this year: SA-26:30 concerned setugid binaries and was fixed in 15.1 before its release, and SA-26:47, a kernel stack disclosure, did not affect 15.1 at all. A host that runs Linux binaries patches its kernel like any other.
The userland is Rocky's. Its fixes reach the jail when the port is updated.
No speed is claimed here. The Foundation says programs run "as fast, and sometimes even faster" than on Linux, and gives no figure, so neither does this piece.
The bill
On most teams today, the one binary that ships only for Linux decides the operating system for the whole machine, or it gets a virtual machine of its own with a second kernel and a second patch cycle. That is the default, and it is rarely costed.
Here is what it costs. A complete distribution kept alive for a single program, with its own release cadence and its own idea of where the network configuration lives. Or a virtual machine with a kernel of its own, for a daemon that wanted a socket. In both cases the service runs with the full kernel interface its distribution happens to allow, unless somebody remembered to write the filter.
On FreeBSD it costs one package for the userland, one user, one jail block of a dozen lines and two resource limits, and the service ends up in a room with one slot in the wall, on a kernel where io_uring, user namespaces, unprivileged bpf and userfaultfd do not exist.
The layer began because somebody wanted to play Doom. Thirty-one years later the same code keeps a vendor's precious Linux binary at a safe distance from everything it does not need.