Vivian Voss

rctl Limits a Jail in One Line

technical beauty scope freebsd rctl jails

A build server runs two jails side by side. One is called ci and compiles whatever the developers push; the other is called web and serves the company's documentation, which nobody notices until it stops. On Tuesday someone pushes a branch with make -j32 in it, and the compiler in ci starts hogging memory like a lodger who has found the biscuit tin. The pages from web begin to arrive slowly, then not at all.

The administrator types one line:

rctl -a jail:ci:vmemoryuse:deny=8g

From the next allocation onwards the kernel refuses ci any address space beyond eight gigabytes, counted across every process in the jail. The compiler's next request for memory comes back with an error and the build fails, which is precisely the right thing for a build that wanted more than its share, and web carries on serving pages. Nothing was restarted and no daemon was asked; the rule went into the kernel, and the kernel did the rest.

rctl came from Edward Tomasz Napierała, under sponsorship from the FreeBSD Foundation. His first commit to kern_rctl.c, on 30 March 2011, is two words long, "Add rctl.", and the command appeared in FreeBSD 9.0. It has been doing this job quietly ever since.

One line, four kinds of tenant

A rule has a fixed shape, which rctl(8) gives as subject:subject-id:resource:action=amount/per. The subject is the thing being limited, and there are four of them: a process, a user, a login class from login.conf and a jail. The resource is what is being counted. FreeBSD 15.1 counts twenty-five of them. Resident memory and swap are on the list, and so are open files, processes, threads, CPU percentage and the rate of reads and writes on the file systems, with the System V message queues and semaphores thrown in for the software that still uses them.

The action says what happens when the amount is reached. deny refuses the allocation, log writes a warning to the console, devctl hands an event to devd(8) so that a script can react, a signal such as sigterm goes to the process, and throttle slows a jail's reads and writes down to the rate the rule allows. The optional /per decides who shares the amount: loginclass:students:maxproc:deny=50/user gives each student fifty processes, where the same rule without /user would give all of them fifty together.

Rules that ought to survive a reboot go into /etc/rctl.conf, one per line in exactly the same format, and the system reads them when it goes multi-user. To see what a jail is using right now, rctl -hu jail:web prints every counter in human units. That is the whole interface, and it pulls its weight.

Where the counting happens

Underneath the command sit two parts of the kernel. racct keeps the accounts: whenever a process allocates memory, opens a file or forks, the kernel adds the amount to a counter for that process, for its user, for its login class and for its jail, all in the same call. rctl keeps the rules: each process carries a list of the rules that apply to it, and before the allocation goes through, rctl_enforce() walks that list and compares what is available with what is asked for. A deny turns into an error for the system call that asked: vm_mmap.c checks RACCT_VMEM before it maps anything, and a refused request returns ENOMEM, which a well-behaved program handles and a badly behaved one reports. Resident memory is the one resource handled differently. The kernel cannot refuse a page that a process merely touches, so for memoryuse the page daemon in vm_swapout.c compares each process with what its rules leave available and moves pages out of memory until it fits.

The counting happens where the resource is handed out, which is why nothing has to poll and nothing can arrive late. There is no daemon watching from outside and no file system of control files to keep tidy. Counted without comments and blank lines, kern_racct.c, kern_rctl.c, their two headers and the rctl command come to 3,314 lines in FreeBSD 15.1.

rctl on FreeBSD 15.1 3,314 lines of code racct, rctl, headers and the command 25 resources 20 in FreeBSD 9.0 4 kinds of subject process, user, login class, jail 0 syntax changes since FreeBSD 9.0

Every system call that touches a rule asks priv_check() for a privilege of the PRIV_RCTL family first. Inside a jail, prison_priv_check() decides, and the RCTL privileges are missing from its list of things a jailed root may do, so the request ends in its default branch: "In all remaining cases, deny the privilege request." The tenant in ci can be root in every other sense and still cannot take the line off.

A jail inside a jail

The jail gave FreeBSD isolation in March 2000, as a single object in the kernel. With FreeBSD 8.0, James Gritton made it hierarchical: set children.max on a jail and the processes inside may create jails of their own. jail(8) sets out the rule of inheritance plainly: "Jailed processes are not allowed to confer greater permissions than they themselves are given." A child of the jail hosting appears in the base system as hosting.alice, and a child of that as hosting.alice.test.

rctl was written for a kernel that already had this hierarchy, and it follows it in two places. When racct adds an amount, it walks up the chain of parent jails through pr_parent and adds the amount to every one of them, so rctl -hu jail:hosting shows the consumption of the jail and of everything nested inside it. When rctl adds a rule for a jail, it binds the rule to every process whose own jail, or any ancestor of it, is the subject. A rule on hosting therefore reaches every process in hosting.alice and in hosting.alice.test, and none of them can remove it.

In practice that gives a hosting provider a no-nonsense way to work. One line on the parent sets the figure that every jail beneath it is held to, each counted on its own, and the parent's own processes are counted against the whole family. A customer who needs a different figure gets a line of their own:

One rule on the parent reaches every jail inside it host rctl -a jail:hosting:vmemoryuse:deny=4g jail hosting limit 4 GB · counter: its own processes and every child mail relay a process of hosting itself jail hosting.alice rctl -a jail:hosting.alice:vmemoryuse:deny=2g held to 4 GB from above and to 2 GB by its own line jail hosting.alice.test bound by both rules above it usage counted upwards
rctl -a jail:hosting:vmemoryuse:deny=4g
rctl -a jail:hosting.alice:vmemoryuse:deny=2g
rctl -a jail:hosting.alice:pcpu:deny=50

The parent jail may run its own processes next to its children, a mail relay or a monitoring agent, and they are counted like everything else. The child gets a smaller box inside the larger one, never a larger one.

The same job, assembled

Linux wanted what the jail offered and built it in separate parts. Mount namespaces came in 2002, the others one at a time over the following eighteen years, and resource control came along as a separate mechanism, cgroups, first released with Linux 2.6.24. Its own manual page, cgroups(7), is admirably frank about what happened next: "the development of these controllers was largely uncoordinated, with the result that many inconsistencies arose between controllers and management of the cgroup hierarchies became rather complex." A second version was started in Linux 3.10 and made official in 4.5, and systemd 258 has since removed support for the first one entirely.

cgroups v2 does a fair bit more than a flat list of rules. It has a weighted hierarchy in which a parent's limit binds the sum of its children, delegation of subtrees to unprivileged users, soft limits such as memory.high and pressure information that tells an administrator how long tasks have been waiting. It also has to stand apart from the thing it limits. A container on Linux is a set of namespaces plus a cgroup, and something outside the kernel, a runtime such as runc or a service manager such as systemd, creates the group, writes the limits into files such as memory.max and moves the processes into it. v2 adds a rule of its own for the shape of the tree, which cgroups(7) calls the "no internal processes" rule: apart from the root, processes may live only in leaf groups, and a group that has children must leave its processes in a subgroup called something like leaf.

The code follows the history. In Linux 7.3, kernel/cgroup comes to 8,830 lines without comments and blank lines once its four cpuset files are set aside (FreeBSD keeps cpuset separately as well); the memory controller in mm/memcontrol.c adds 4,031 and the block controller in block/blk-cgroup.c 1,361. That is 14,222 lines for the files that carry the jobs rctl does, beside 3,314 on FreeBSD. The larger tree buys the weighted hierarchy and the pressure figures, and it costs every administrator a second structure to learn, kept beside the container it governs.

Where the limit lives FreeBSD Linux the jail: one object in the kernel already there since FreeBSD 4.0 one rctl line names the jail or one line in /etc/rctl.conf kernel checks it at every allocation root inside the jail cannot remove it 3,314 lines with the command namespaces for the isolation added in parts, 2002 to 2020 a separate cgroup tree for the limit v2, processes only in the leaves a runtime joins the two runc or systemd creates the group, writes memory.max into a file, moves the processes in · 14,222 lines

On FreeBSD the question of where to put the limit never comes up. The jail is already there, and the limit is a property of it.

The sheet

Efficiency. FreeBSD 15.1 (releng/15.1, commit 95719d8), lines without comments and blank lines: sys/kern/kern_racct.c 855, sys/kern/kern_rctl.c 1,678, sys/sys/racct.h 156, sys/sys/rctl.h 103, usr.bin/rctl/rctl.c 522, together 3,314. The command ships with the base system and runs no daemon; GENERIC already carries the kernel options RACCT and RCTL. For comparison, Linux 7.3 (commit c03c1a1): kernel/cgroup 11,990, or 8,830 without the cpuset files, mm/memcontrol.c 4,031, block/blk-cgroup.c 1,361.

Security. Of the four questions: the rctl command needs root, and only root outside a jail, because prison_priv_check() denies every RCTL privilege inside one; it takes no data from the network; it parses one rule string of five fields; and the enforcement runs in the kernel, at the allocation, so there is no long-running privileged process to attack. No FreeBSD security advisory carries rctl or racct in its name.

Longevity. First commit on 30 March 2011, in FreeBSD since 9.0, 52 commits to kern_rctl.c in fifteen years, maintained in the base system ever since.

Stability. The rctl(8) page of FreeBSD 9.0 lists four subjects and twenty resources, and its examples, rctl -a user:joe:vmemoryuse:deny=1g and rctl -hu jail:www, stand word for word in the page for 15.1. Five resources and the throttle action have joined since; the syntax has not changed by a character, and /etc/rctl.conf reads the same format it read in 2012.

Measured on 13 November 2026: lines counted with a script that strips C comments and blank lines, over freebsd-src releng/15.1 and torvalds/linux v7.3; resources counted in rctl(8) of FreeBSD 9.0 and 15.1; commits counted on the history of sys/kern/kern_rctl.c; the kernel paths read in kern_racct.c and kern_rctl.c, with the jail check in kern_jail.c and the memory paths in vm_mmap.c and vm_swapout.c.

The limit

racct is switched off in the default kernel. GENERIC carries the option RACCT_DEFAULT_TO_DISABLED, and kern.racct.enable is a loader tunable. A first rule therefore needs kern.racct.enable=1 in /boot/loader.conf and a reboot; after that every rule takes effect at once.

A rule on a parent jail holds each nested jail to its figure on that jail's own count. The total of the whole family binds the parent's own processes. Where the sum across all children must stay under one ceiling, each child gets a line of its own.

Some counters are estimates. rctl(8) says so under BUGS: the read and write operations are counted in the file system layer and are "only approximations", and a ceiling on resident memory is enforced by paging out, which can send a busy machine into thrashing. A ceiling on address space or swap, or a devctl action that lets a script decide, is the gentler tool.

There is no weighted share and no pressure gauge. A rule is a ceiling, and pcpu caps a jail as a percentage of one core. Proportional weights between jails and the stall figures of cgroups v2 have no counterpart; the ceiling is what most hosts on a shared machine ever needed.

The limit is one line on the jail, and nobody inside the jail can take it down.