Vivian Voss

What Was Already Watching

freebsd blacklistd fail2ban ssh

Every hardening guide gives the same answer to the same question. The server is reachable on port 22, the log fills with strangers trying root with the password root, and somebody sensible says: install fail2ban.

It is a good answer. It works, it is portable, it has been the reasonable thing to do for two decades, and this measurement does not end with it looking foolish.

What I wanted to know was what it costs, and what else was available. So I weighed it, put a stopwatch on it, and then went looking at what the machine had been doing about the problem before I arrived.

How this was measured

FreeBSD 15.0-RELEASE-p13 on my own hardware, in the jail test_and_benchmarks, rolled back to its golden snapshot before the run and again afterwards.

One honest caveat about the bench. The host was not idle. Load average stood at 12.92 throughout, because a service of mine was doing real work and I let it. Byte counts and file counts survive that untouched; sub-millisecond timings would not, so none appear below. Where a time is quoted it is a whole-second figure around a whole chain.

The jail started at 349,085 kilobytes with one package installed, that package being pkg itself.

What the standard answer weighs

pkg install py312-fail2ban, and one package became fourteen. Disk went to 450,785 kilobytes.

The tool itself is 3,451,627 bytes by the catalogue. The interpreter it requires is 229,854,412. Sixty-seven to one, between the thing you asked for and the thing that had to come along. Running, the daemon holds 26,960 kilobytes resident.

The package lays down 178 configuration files, of which 103 are filter definitions and 66 are actions. The filter for the job in question, filter.d/sshd.conf, is 139 lines carrying 27 regular expressions.

Twenty-seven patterns, to recover from a line of English text a fact that sshd had in a register a moment earlier. It knew the username. It knew the source address. It knew the attempt had failed. Then it formatted all of that into prose, wrote the prose to a file, and a second program with a hundred-megabyte interpreter set about parsing it back.

fail2ban is doing the only thing the available interface allows. The prose is the interface.

Same failed login, two routes to the block VIA TEXT VIA SOCKET sshd formats it into prose syslog writes the file 229 MB interpreter loads 27 patterns, parses it back counter reaches maxretry action calls the firewall five steps, three of them text eight conditions must hold sshd reports over a socket blacklistd counts to three, writes the pf table pf drops it for 24h three steps, no text at any one line of configuration

The chain, and the stopwatch on it

Fair is fair, so here is the timing. Three failing lines into the watched file, maxretry at three, and the action fired 1.01 seconds later. Anybody who tells you fail2ban is slow should be shown that number.

The count that matters is the other one. Eight things have to be right for that second to arrive: the daemon up, syslog writing, the line reaching the configured path, the format matching, one of the 27 expressions firing, the timestamp inside the window, the counter reaching the threshold, and the action able to reach the firewall. A break in any one produces the same visible result as a working system, because nothing gets written down to record that nothing happened.

I broke two of them myself while building this bench, which is the most honest endorsement of the point I can offer. The first run counted nothing, because my timestamp carried a leading zero where syslog writes a space. The second reported a ban in 0.0007 seconds, because the dummy action had already written its target file at startup.

The third turned up unasked. Inside the jail the clock says 06:21:01 UTC; on the host, the same second, 08:21:01 CEST. A log written on one side of that line and parsed on the other sits two hours outside every window it is judged against.

The chain with no text in it

Now the same job done the other way, on the same machine, and this is the part I would ask a sceptical reader to look at properly.

Here is the whole of /etc/blacklistd.conf on the host, comments removed:

[local]
ssh   stream  *  *  *  3  24h

Seven fields. The service, the socket type, three wildcards for owner and name and address family, then the number of failures that count and how long the block lasts. That single line is the entire configuration of the mechanism.

What happens underneath it is a chain of four steps, and none of them involves text. sshd fails an authentication and reports it over a socket to blacklistd, which is a function call rather than a sentence. blacklistd counts, and on the third failure it adds the address to a pf table. pf drops everything from that table for twenty-four hours. Nobody formats anything, nobody parses anything, and there is no window in which a mismatch between two versions of a log format quietly breaks the arrangement.

On this host the result is visible in two commands, one addressed to the packet filter and one to blacklistctl. The packet filter holds exactly one table, bruteforce, with 7 addresses currently in it. And blacklistctl dump -a shows 135 entries in the ledger behind it, the most recent from yesterday evening.

So: seven fields of configuration, against 27 regular expressions, 178 configuration files, a 229-megabyte interpreter and a text format that both ends have to keep agreeing on. Same outcome, same port.

What was already running, before anything was installed OpenSSH 10.0p2 PerSourcePenalties on by default refuses for up to 10 min socket blacklistd 47,392 bytes counts to three one config line table pf: bruteforce 7 addresses drops for 24h ledger of 135 pfsync second machine same list after failover Every piece here shipped with the base system. Nothing was installed for it, and nobody had configured it.

Two more pieces sit around that chain, both of them already running.

sshd also refuses repeat offenders itself. OpenSSH 10.0p2 carries PerSourcePenalties, and the running configuration reports this without a line of it in any file I wrote:

persourcepenalties crash:90 authfail:5 noauth:1
  grace-exceeded:10 refuseconnection:10 max:600 min:15

Three failures reach the fifteen-second threshold and the source is refused for up to ten minutes. That happens before blacklistd has been asked for an opinion.

And when the machine gets a second one beside it, pfsync carries the state table and the block list across, so a failover hands an attacker no clean sheet. People forget that piece: a blocklist living inside one appliance is lost the moment the appliance is.

What of that works on Linux

This is the question that decides whether any of the above is worth a reader's time, and it deserves a straight answer rather than a flattering one.

The penalties work on Linux. They are a feature of OpenSSH, not of any operating system, and they arrived in 9.8 in 2024. Debian 13 ships them and they are on by default there too. Anybody running a current distribution already has that half.

The filter half works on Linux as well. nftables has dynamic sets with timeouts, and a rule can add an offending source to such a set with update when it exceeds a rate. That is the same idea as an overload table, it performs well, and people have been writing about it for years.

What does not exist on Linux is the middle one. There is no blacklistd in any mainline distribution, which means the pattern where the authenticating service reports its own verdict to a separate enforcement daemon has no home there.

So the honest scoreboard is: two of the three alternatives are available to a Linux administrator today. And almost nobody uses them.

That gap between available and used is the whole of what I would want somebody to take from this. Ask an experienced Linux administrator how to stop SSH brute force and the answer is fail2ban, because that is what every guide says, every distribution packages, and every colleague has done. The nftables approach lives in blog posts. The penalties arrived quietly in a release note.

Same four pieces, two places to keep them FREEBSD LINUX ADDED ON fail2ban, in ports, optional fail2ban, what guides say SERVICE sshd penalties, on by default sshd penalties, on by default REPORTING blacklistd, base system no equivalent anywhere FILTER pf, base system nftables, dynamic sets Three of the four boxes are green on both sides. The difference is which one you find first.

On FreeBSD the same three answers are the documented default. blacklistd has an rc script and a manual page in the same tree as the daemon that feeds it, and pf carries the table syntax in the same manual as everything else. All of it was released and tested together by the people who wrote both ends.

That is the difference worth paying attention to, and it is not about capability. Both systems can do this. On one of them the integrated answer is what you find when you look; on the other it is what you find after you have already installed the parser.

Three questions, one minute

Whichever system this is read on, the measurement reduces to three questions that take about a minute together.

Are the penalties on?

sshd -T | grep persourcepenalties

A line means the daemon is already refusing repeat offenders. Nothing means the OpenSSH is older than 9.8.

Is anything already keeping a list?

service blacklistd status
blacklistctl dump -a | wc -l

On FreeBSD that may well surprise you. On Linux it will report that there is no such service, which is itself the answer.

And the one I would ask first on any machine where a log watcher was installed and then forgotten:

fail2ban-regex /var/log/auth.log \
    /usr/local/etc/fail2ban/filter.d/sshd.conf

It runs the real log against the real filter and counts the matches. Hits mean the chain is intact. A count of zero means it is broken somewhere, and it has been broken for as long as nobody ran this command, because a watcher that matches nothing behaves exactly like a watcher with nothing to do.

The limit

Three deductions against the above.

The comparison is not like for like. Penalties and blacklistd defend the daemon that reports to them. fail2ban defends anything that writes a line, which is why 103 filter definitions exist, and why a web login form has no other option on either system.

The 135 entries prove that something is counting. They do not prove it prevented an intrusion, and nobody can measure the break-in that did not happen.

And the penalties are young. Anybody who examined this question three years ago examined it correctly and got a different answer, which is worth remembering before treating any of this as settled.

The point

The claim under examination was that fail2ban is the answer to brute force on SSH. It is an answer, and on a machine with an application that authenticates in its own code it is the only one.

For the specific case of a daemon that already knows the verdict, it is the most expensive of the four, and the only one that fails silently. It costs a hundred megabytes of interpreter and twenty-seven patterns to reconstruct a fact that was never lost, only formatted. The alternative on this machine is seven fields in one line.

Wherever two programs were built by people who never had to agree on anything, one ends up writing prose and the other ends up parsing it. Somebody then maintains the regular expressions in between for as long as the arrangement lasts. Where they were built together, the second program is simply told.

On this machine the answer was a list of a hundred and thirty-five, kept by a program in the base system, which nobody had configured and nobody had thanked.