Eleven Lines for a Dozen Jails

Unix Universe ■ the receipt

Here is the whole of /etc/rc.conf on a FreeBSD machine I run. It carries a dozen jails, several websites, a firewall and a storage pool. The addresses below are from the documentation ranges, because publishing the real ones would be a curious way to spend a Monday.

hostname="host.example.com"
ifconfig_ext0="SYNCDHCP"
ifconfig_ext0_ipv6="inet6 2001:db8:a3f2:9e01::1 prefixlen 64"
cloned_interfaces="lo1"
ifconfig_lo1_name="jail0"
sshd_enable="YES"
jail_enable="YES"
pf_enable="YES"
zfs_enable="YES"
ntpd_enable="YES"
sendmail_enable="NONE"

Eleven lines, 285 bytes. sh -n /etc/rc.conf passes, and it passes because the file is a shell script that assigns variables, which the startup machinery then sources. No configuration format comes into it anywhere.

That fact does more work than it looks.

The line you type twice

Bring an interface up by hand:

ifconfig ext0 inet 192.0.2.1/24 up

Now make it survive a reboot:

ifconfig_ext0="inet 192.0.2.1/24 up"

Those are the same arguments, character for character, moved from the command line into a file. No translation step sits between them. If a configuration works when you try it, you keep it by copying it.

And when a script has to do the keeping, there is a third place for the same string:

sysrc ifconfig_ext0="inet 192.0.2.1/24 up"

sysrc reads, writes and removes variables in rc.conf without a text editor and without a templating step. It arrived in FreeBSD 9.2, written by Devin Teske, and it exists precisely because a shell file that a machine has to edit deserves a tool that will not corrupt it.

One string, three places. Nobody had to invent an intermediate representation, because the intermediate representation is the command you already know.

One pattern, one hundred and thirty-three services

Open /etc/defaults/rc.conf and count. It holds 475 variables, of which 133 end in _enable and 85 in _flags. Those suffixes are the whole interface, and they are the same two suffixes everywhere.

sysrc nginx_enable=YES
service nginx start

Substitute any of the other hundred and thirty-two and the shape does not change. Somebody who learned the pattern once can switch on a service they have never heard of, correctly, without opening the manual. The manual is still there and still worth reading, but it is not the price of admission.

The service scripts themselves are the same story at a smaller scale. There are 171 of them in the base system, and /etc/rc.d/sshd is 87 lines of shell. Those 87 lines are the work itself, and they read start to finish while the kettle boils.

Three thousand lines, and a paper from 2001

The machinery underneath is /etc/rc.subr with 2,870 lines and /etc/rc with 152. Three thousand and twenty-two lines of shell start the network, the firewall, the storage layer, the jails and everything else, and any of it can be read by the person operating the machine.

This was a decision, and it has an author. The framework came from NetBSD, contributed by Luke Mewburn in the late nineties, and he wrote up the reasoning for USENIX in 2001. FreeBSD adopted it, and the copyright header in rc.subr still carries the NetBSD Foundation's name three decades later, which is a pleasant thing to find in a file you are debugging at half past two.

The design goal was ordering and dependency handling for startup scripts. What came out of it was something broader: a system whose entire runtime configuration is expressible in variables, and whose startup logic is expressible in shell.

From one machine to the fleet

Here is where the shape pays off, and it is the part people miss when they call this arrangement old-fashioned.

Because the configuration is a text file and the administration is shell, the same language reaches from the keyboard to the jail to the whole fleet without changing form. Into a jail from the host:

service -j www nginx restart

Across machines, with nothing installed on the far end:

for h in host1 host2 host3; do
    ssh $h 'sysrc nginx_enable=YES && service nginx start'
done

No agent on the targets. No control plane. No template renderer turning one dialect into another. The remote command is the local command, and the file it writes is the file you would have edited.

That is worth being precise about, because it is easy to overclaim. A loop over three hosts is not configuration management. It has no state reconciliation, no idempotency guarantees beyond what the commands themselves provide, no inventory and no reporting. Ansible, Puppet and their relatives exist because those things matter once a fleet gets large enough or a team gets big enough, and they solve real problems that a loop does not.

The observation is narrower. A great many installations that reach for a control plane are running fewer machines than a person has fingers, and on this platform the shell already carries them.

For anyone who likes it more comfortable

The shell is doing rather well at this point, with a set of tools that have had four decades to settle into their job. For anyone who would like the same work with fewer keystrokes and something to look at while doing it, there is a project worth watching.

Sylve is a unified management platform for FreeBSD, sponsored by the FreeBSD Foundation and built by Hayzam Sherif at Alchemilla: a web interface over bhyve virtual machines, jails, ZFS, networking and firewalling, with clustering and host-to-host migration on the way. Proxmox is the obvious comparison and the project makes it themselves.

The interesting part is how it works underneath. The quarterly status report describes the ZFS layer as a Go wrapper around the command line tools, and the platform reads the JSON output that the base system already produces. So the panel sits on top of the same commands and the same files a person would use by hand. The truth about the machine stays where it was: in rc.conf, in jail.conf, in the ZFS properties.

That is the difference worth naming, and it has nothing to do with taste in interfaces. A management layer that reads and writes the system's own configuration can be installed, used and removed, and the machine carries on. A management layer that keeps the authoritative state in its own database becomes load-bearing the moment you rely on it, and taking it away is a migration.

Structured output stops being a curiosity at exactly this point. netstat --libxo json and the JSON that ZFS emits are not there to look modern; they are the reason somebody can build a panel without inventing a parallel source of truth.

The tools that kept growing

None of this would matter if the commands underneath had stood still. They did not.

ifconfig(8) on FreeBSD 15 has a manual page of 2,527 lines documenting 297 parameters and subcommands. From that one command you create bridge, epair, gif, lagg, tap, tun, vlan, vxlan and wg interfaces. WireGuard, in other words, is configured by a program that first appeared in 4.2BSD in August 1983, using the same grammar it had then, and the line you type to do it is the line you put in rc.conf.

The same holds along the row. route for routes, arp and ndp for neighbours, netstat for state. Four questions, four names, and no object model to learn before the first one works. The same commands, with the same grammar, run on FreeBSD, macOS, OpenBSD, NetBSD and illumos. Knowledge learned once travels with the person, and it keeps working after a change of platform.

The limit

Four honest deductions.

A shell file has no schema, and that cuts both ways. Type sshd_enabl=YES and sysrc writes it without a murmur; the service stays off, and nothing tells you why until you go looking. The information to catch it exists, since sysrc knows the valid names well enough to answer unknown variable when you ask for one that is not there. It simply does not apply that knowledge on the way in. The check that does exist is service sshd rcvar, which prints what the system actually believes, and it is worth the habit.

A single command with 297 parameters is not, by any reading, one thing done well. ifconfig is a multiplexer that grew, and the only reason it does not feel like one is that the grammar stayed consistent while the surface expanded.

The structured output is not uniform. netstat and arp speak libxo; ifconfig --libxo json produced nothing on the machine I tested it on. Anyone building tooling on top should check each command, because the base system is uneven here and quietly so.

And the portability argument only pays where the fleet is mostly BSD. Somebody who runs three FreeBSD hosts and thirty Linux ones needs both vocabularies regardless, and for them the second one is not optional.

The bill

The industry has spent a decade building a layer whose job is to turn one description of a machine into another description of a machine. YAML into unit files, HCL into API calls, a template into a config, and a control plane to keep track of which version of which rendering landed where.

Every one of those layers was bought for a reason, and at scale most of them earn their keep. The question worth asking is where the reason stops applying. On the machine described at the top of this piece, the description of the system is the system's own language, the tool that edits it is a shell script, and the thing that applies it across hosts is a program from 1995 that everyone already has installed.

A configuration you can read in eleven lines is one you can also diff between two machines, put in version control without a converter, and repair from a rescue shell when nothing else on the box will start.

Somebody chose this shape in 1997 and wrote down why. The interesting part is that it never needed replacing to keep up.

← Unix Universe