Vivian Voss

There Is No Linux Admin

linux freebsd sysadmin hiring

The advert says: Linux Administrator. Three or more years of experience. Sometimes a distribution appears in brackets, sometimes not, and when it does not, nobody in the room notices that anything is missing.

Somewhere else in the same building, a decision is taken that will outlive most of the people taking it. Somebody proposes FreeBSD for a particular job, and the answer arrives before the argument does: we have nobody in-house for that.

Both sentences rest on a category. It is worth asking whether the category exists.

Four Layers, and What They Say

An operating system, from the perspective of the person administering it, is four things sitting on top of a kernel. A C library. A set of core utilities. An init system. A way of installing software.

Alpine describes itself in one line: built around musl libc and busybox, with its own package manager called apk and the OpenRC init system. Fedora runs glibc, GNU coreutils, systemd and dnf. Not one of those four is the same. The kernel is.

Now put FreeBSD and macOS beside them. FreeBSD carries its own C library and its own utilities, both descended from Berkeley. macOS carries Libsystem and a userland with the same Berkeley ancestry, which is why ls -l on a Mac behaves like ls -l on FreeBSD and unlike ls -l on Fedora. Init and packaging differ, launchd against rc.d, App Store against pkg.

Four layers, four systems SYSTEM C LIBRARY UTILITIES INIT PACKAGES Alpine musl BusyBox OpenRC apk Fedora glibc GNU coreutils systemd dnf FreeBSD BSD libc BSD utilities rc.d pkg macOS Libsystem BSD utilities launchd App Store Alpine and Fedora, one job title: 0 of 4 layers shared FreeBSD and macOS, different worlds: 2 of 4 layers shared And the two shared layers are the ones an administrator touches every minute of the working day.

Count the shared layers. Alpine and Fedora share none of the four. FreeBSD and macOS share two, and those two are the ones an administrator touches every minute of the working day.

The category that hires people is drawn around the layer nobody administers.

Where the Divergence Actually Bites

That would be an amusing table and nothing more, if the differences stayed in the specification sheet. They do not.

awk is a POSIX tool with a POSIX specification. Fedora and Arch ship GNU awk. Debian and Ubuntu ship mawk. Alpine ships the BusyBox implementation. For years, a bracket expression that matched digits worked on one and returned nothing on the other, and asking gawk for the length of an array returns the number of elements while mawk treats the same expression as an uninitialised scalar and answers zero. Not an error. Zero.

/bin/sh is worse, because everybody assumes they know what it is. On Debian and Ubuntu it is Dash. On Fedora and Arch it is Bash. On Alpine it is BusyBox ash. Every Bash convenience a working administrator has in their fingers, process substitution, double brackets, brace expansion, arrays, fails on Dash, and fails quietly.

#!/bin/sh
targets=(alpha beta gamma)     # Fedora: /bin/sh is Bash, works
echo "${targets[0]}"           # Debian: /bin/sh is Dash
                               #   "Syntax error: ( unexpected"

Ubuntu maintains an entire wiki page listing the incompatibilities, which is a curious document to need for a shell that is supposed to be the common denominator.

No error. No warning. The script that works on one distribution silently stops working on another, and the only clue is an empty terminal.

Somebody who has administered Debian for six years and is handed Alpine has a different libc, a different shell, different utilities, a different init and a different package manager. Somebody who has administered FreeBSD and is handed macOS recognises most of what is under their hands. The advert would hire the first as a match and reject the second as a stranger.

The Canon That Was Replaced

Here the argument turns, and it turns against the assumption rather than against the people who hold it.

An administrator operates a canon: services, network, firewall, packages, logs, storage, users, boot. Eight things. Learn them on one system and you have learned the shape of the work everywhere.

In the Linux world, that canon was replaced while people were using it. systemd became the default init in Debian 8, and the same move happened across the ecosystem. nftables became the default firewall framework in Debian 10, displacing the iptables syntax that a generation had committed to memory. ifconfig and route gave way to ip, netstat to ss, syslog to journald, yum to dnf. Each change had reasons, and I will come to them.

The canon, replaced while in use JOB LINUX 2010 LINUX TODAY FREEBSD, BOTH YEARS services SysV init scripts systemd (Debian 8) rc.conf firewall iptables nftables (Debian 10) pf interfaces ifconfig, route ip ifconfig sockets netstat ss netstat logs syslog journald syslog packages yum dnf pkg, ports A Debian manual from 2010 now walks into commands that are absent, deprecated or silently shimmed. The FreeBSD Handbook from the same era still describes the system in front of you: same file, same syntax, same manual page, every word of it still true this morning.

The consequence for the person doing the work is unsentimental. A Debian manual from 2010 will now walk an administrator into commands that are absent, deprecated or silently shimmed. The FreeBSD Handbook from the same era still describes a system where services are configured in rc.conf and the firewall is pf, with software arriving through ports and packages. Same file, same syntax, same place, same manual page, and every word of it still true this morning.

Which inverts the sentence that started this piece. The risk to a staffing plan is not the system that has kept its canon for two decades. It is the ecosystem that has replaced its canon three times, and where the manual on the shelf ages faster than the person who read it.

The Limit

The strongest version of the opposing case deserves stating properly, because the replacements were not vandalism.

systemd solved real problems. Parallel startup, dependency ordering that a machine can verify, cgroup integration, and one declarative service description instead of several hundred lines of hand-rolled shell per daemon, each with its own idea of what a PID file is. nftables folded four separate frameworks into one, which to my eye is the better design and to somebody else's may not be. ip genuinely does more than ifconfig ever could. Anybody who claims these were changes for their own sake has not maintained a boot sequence.

Whether the thing that starts services should also handle logging, name resolution, network configuration and device management is a separate argument, and a sharp one, with a good deal to be said against it. It belongs somewhere other than here. What matters for this piece is narrower: the replacement happened, and it happened to people who had already learned the old way.

Some of them also share a great deal more than the kernel, and the advert is not entirely wrong about that. Debian and Fedora agree on the filesystem layout, the permission model, the process model, the network stack, and the shape of nearly every configuration file. Somebody who knows Ubuntu can run Debian by Thursday, which proves less than it sounds, because Ubuntu is built from Debian and inherits the lot.

Walk away from the parent and the ground starts moving. Put that same Debian administrator in front of Gentoo and packages arrive as source, compiled on the machine against a set of feature flags that change what the finished binary contains at all; the init system is not the one they know either. Call it half the working day unfamiliar. Slackware costs more than that, because there is nothing resolving dependencies for you, a package is a tarball, and a service is an executable shell script in /etc/rc.d that you switch on and off with chmod. Slackware calls that arrangement the BSD-style layout, and it is not being poetic.

The distance from Debian, measured in relearning Ubuntu a dialect: built from Debian, inherits the lot Gentoo source builds, USE flags, different init Slackware no dependency resolution, rc.d scripts, chmod minimal container image no package manager, frequently no shell, an unfamiliar libc What the distance turns on is lineage, not the label: a derivative sits close to its parent, a project with a different mission may share little beyond the kernel and the licence.

Sit with that for a moment. The administrator who has learned Slackware is closer to FreeBSD's rc.d than to the systemd units running on the Fedora machine in the next rack, and the advert files two of those three under the same job title. A minimal container image finishes the demonstration: no package manager, frequently no shell, a libc nobody in the building has linked against. Twenty years of Debian buys you almost nothing in there.

So the market has noticed something real and has drawn the boundary in the wrong place. The same word covers a dialect and a foreign language, and the advert never says which one is waiting.

And the transfer has limits in the other direction too. The fundamentals move across in days: services, packages, firewall, logs. The depth does not. Jails, ZFS beyond creating a dataset, Poudriere, the particular logic of a base system that is built and released as one thing, those take weeks to months, and anybody who tells you otherwise is selling something.

Then there is the part that has nothing to do with skill. Vendors ship Linux packages, Linux drivers and Linux support contracts, and for a great deal of commercial software that is the whole of the list. If the sentence we have nobody in-house for that actually means we would carry that alone at three in the morning with no vendor to ring, then it is not prejudice. It is an accurate description of a risk, and it deserves a better hearing than the version that hides behind a job title.

So the advert names a kernel family and the meeting treats it as a description of a person. What follows is a decision about architecture, taken on the strength of a category that does not survive contact with four rows of a table, and taken by somebody who never intended to decide anything about architecture at all.

Call it the staffing veto. It arrives early, it needs no evidence, and it is almost never revisited, because by the time anybody could check it the system has been built the other way and the checking would be expensive.

There is a Debian admin. They read Ubuntu as a dialect, relearn half the job on Gentoo, more than half again on Slackware, and stand in front of a minimal container image with twenty years of experience and nowhere to put it. That is a real person, and you can hire them. What there is not, and never was, is a Linux admin.