Vivian Voss

The Bug in the Gap

freebsd security containers openssh

Last Friday ended on a question. When one of these surfaces finally gives, who patches it, and how fast can you trust the patch? The honest way to answer it is to pick a morning everyone in this trade remembers, and watch what each kind of system actually did with it.

The morning is 1 July 2024. Qualys publishes regreSSHion, CVE-2024-6387: a signal-handler race in OpenSSH's server that, on glibc-based Linux, hands an unauthenticated stranger the ability to run code as root. This is the OpenSSH on every server you own. The bug is not the interesting part; bugs arrive like weather. The interesting part is the answer, and how long it took to become one you could trust.

On FreeBSD the answer was a single advisory, FreeBSD-SA-24:04, and six supported branches corrected between 08:22:13 and 08:27:53 UTC the same morning. That is a window of five minutes and forty seconds: one team, one tree, one commit typed once and used six times, done before most of the people affected had finished their coffee. On a fleet of Linux machines the same fix was equally real and equally arriving, but it arrived as a week of separate advisories, each distribution on its own calendar and its own maintenance window, like guests answering an invitation whenever it happens to suit them. Same upstream patch. Two rather different amounts of time before you could say, and mean it, that the thing was handled.

Hold on to that gap between one coherent answer and a scattered one, because it is the subject of this last piece. And it turns out the gap is not only where the answer lives. It is where the bug lived in the first place.

The Breach

Take the container break-outs disclosed in November 2025 by Aleksa Sarai, a runc maintainer and OCI board member at SUSE: CVE-2025-31133, CVE-2025-52565 and CVE-2025-52881. The mechanism deserves one careful look, because the shape of it is the entire argument. An attacker who can start a container with a crafted mount swaps something innocent, /dev/null or a console device, for a symlink pointing into procfs. runc, doing precisely and only what it was told, bind-mounts the target read-write, and a write meant for a device file lands instead in /proc/sys/kernel/core_pattern or /proc/sysrq-trigger. From inside the container, that is a door to the outside.

Now put each component in the witness box and ask whose fault it is. The kernel? procfs behaved exactly as documented. The namespace layer? It isolated precisely what it promised to isolate. runc? It performed a mount it was validly instructed to perform. The image? Crafted mounts are a supported feature. Every component, cross-examined alone, has a complete alibi. And yet the prisoner escaped. The break-out happened in the one place nobody was cross-examining: the seam between them, the writable procfs surface that the kernel, the namespaces and the runtime each assumed one of the others was minding.

Every Part Correct. The Gap Was Not. kernel + procfs behaved as documented namespaces isolated as promised runc mounted as asked the seam: one writable procfs surface, minded by no one The break-out lived in none of the parts. It lived in the gap between them.

That is one seam, held up as a specimen and not a census. The point is never how many there are. It is where they are, and it is always the same place. Never inside a component. Always in the gap.

The Pattern

There is a phrase for this, older than containers: individually correct decisions that produce a collectively incorrect result. The purest specimen has nothing to do with security at all.

In 2020 the Linux kernel gained a new system call, faccessat2. Entirely correct: it fixed a real flag-handling gap in the old one. In early 2021 the GNU C library, version 2.33, began calling faccessat2 first and only falling back on older kernels. Also correct: prefer the better primitive. And the default seccomp profile shipped by the container runtimes of the day did not list faccessat2 among the permitted calls, because a conservative allow-list naturally omits a syscall that did not exist when it was written. Correct a third time. Put the three impeccably correct decisions in one container and a program asking a question as mild as may I read this file is met with a refusal the C library never budgeted for, and builds fail with errors that make no earthly sense. Four parties, every one of them in the right, had between them fitted a locked door where none was wanted. Nobody had ordered a door. That is simply what happens when everybody is minding their own component and nobody is minding the gaps between them.

And the assembled model does not merely manufacture the bug; it scatters the repair to the four winds. Engineers at CIQ, a firm that ships its own enterprise Linux and therefore has a horse very much in this race, went looking for how faithfully a frozen vendor kernel keeps up with upstream fixes. In the RHEL 8.8 kernel, as of May 2024, they counted 4,594 known bugs for which a fix already existed upstream and had simply never been back-ported. Read that number twice. It is not a queue that a brisk fortnight clears; it is a standing condition, and the sentence the researchers set beside it explains why: individual changes are never tested alone, but only as part of a larger kernel. Cherry-picking four and a half thousand fixes across dozens of independently maintained trees is not work someone has been idle about. It is work the architecture makes structurally unaffordable, which is a deeper and more permanent thing than slowness. Whatever else 4,594 unshipped fixes may be, they are not a rounding error.

One Advisory, or a Week of Them FreeBSD one advisory, six branches, 08:22 to 08:28 UTC Linux fleet a week of separate advisories, each distro on its own calendar And 4,594 upstream fixes never reached the RHEL 8.8 kernel at all. Not two work ethics. Two architectures.

The Limit

Two honesties are owed before the turn.

The first is that the container ecosystem earned its place, and none of the above is a brief against it. Portability across wildly unlike hosts, density, a tooling culture that made deployment legible to a whole generation: these are real goods, they are why the model won, and a symlink trick through procfs un-wins none of them. The people who found and fixed these break-outs are the same community that built the thing, which is precisely the point.

The second is that FreeBSD does not get to fold its arms and look serene. Weeks after regreSSHion it shipped its own advisory, CVE-2024-7589: another unauthenticated remote path to root in its own sshd, and the fault lay in FreeBSD's own integration of blocklistd into OpenSSH, a signal handler reaching for code that was never safe to call there. An integrated project that extends a component can plant a fault in the very act of integrating it. So the claim here has never been that one system is without sin. It is narrower, and it survives the confession: one system has fewer classes of bug, and answers the ones it has from a single place. Lower defect density. Not zero. Anyone selling zero has something else to sell you.

The BSD Angle

Now the turn, and it is worth making without flinching.

A FreeBSD jail, from FreeBSD 4.0 in 2000, is one kernel object. The isolation boundary lives inside the kernel's own system-call layer, enforced in the same place the kernel enforces everything else. There is no runtime process interpreting mount instructions to be tricked, no namespace subsystem to be composed with a control-group subsystem to be composed with a procfs convention that three parties read three ways. The break-out this piece opened with needs four parties holding four slightly different pictures of one shared surface. A jail has no seam of that kind for the plain reason that it has no four parties. The boundary is one thing, in one object, in one tree, and a bug cannot hide in a gap that was never opened.

Where a Seam Can Open assembled — four parties namespaces control groups the runtime procfs convention four seams, four places to hide integrated — one object jail: boundary in the syscall layer, one tree, kernel + libc, one commit no seam anywhere A bug cannot hide in a gap that was never opened.

And the tree is the other half of it. src.freebsd.org holds the kernel, the C library, the userland, the firewall and the documentation in a single repository, built together with make buildworld. A faccessat2-shaped gap cannot open between the kernel and libc when the kernel and the libc that calls it are cut from the same commit and tested as one object. There is no daylight between them for a bug to occupy. And when something does break in the open, that same coherence is the whole reason regreSSHion was answered in six branches inside five minutes and forty seconds: not heroics, not faster typists, but the flat mechanical consequence of one team owning the entire tree and therefore being able to fix all of it at once, in one advisory, before nine in the morning. The scattered week and the five-minute window are not two work ethics. They are two architectures, doing exactly what their shapes require.

The Point

Three Fridays, one argument, and it closes here. First the surface a kernel takes on for speed, the async machinery it admits into Ring 0. Then the programmability it takes on for capability, the foreign programs it agrees to run there. And now the seam, where the answer has to live when either of those finally gives. Three faces of one shape: a system assembled from many sovereign parts accretes surface, retrofits its containment, and scatters its repair across every project that must agree before anything is truly fixed; a system built as one integrated tree adds less, plans the capability it does add, and answers from a single place, fast enough that the answer is trustworthy while it still matters.

Which is the question this whole series was ever really about. Not what a server operating system can do. What it will not do to you, at three in the morning, eighteen months after you chose it. You do not get to choose your bugs. You choose your defect density, and it was chosen the day the system was designed, by how it was built and not by how clever its newest part contrives to be. When regreSSHion lands on the integrated system, the difficult morning is over before it began, and nobody writes the incident up, because there is no incident. That is the entire, unglamorous, load-bearing point.

Boring is not the absence of engineering. It is engineering, banked early, paying out on the worst morning of your year. And it was a choice, made by someone, long before it was ever yours.