Vivian Voss

CARP Fails Over Without a Daemon

technical beauty scope freebsd carp high availability

Somewhere tonight there is a pair of servers with one address between them, and a power supply in the first of them that has chosen three in the morning to retire. The clients have only ever known the address, and whether they notice anything depends on how quickly the second machine works out that the first has gone quiet and how cleanly it takes the address over.

On Linux that work belongs to a program. More often than not it is keepalived, a daemon whose profession is noticing that another machine has stopped breathing. It speaks VRRP, the IETF's protocol for exactly this job, and it also checks the services behind the address and drives IPVS, the kernel's load balancer, all from one configuration file. On a fresh Debian 13 it arrives with sixteen further packages and 63.6 MiB, Perl among them, because its SNMP support wants a word.

On FreeBSD the same arrangement is two lines long and runs no process at all. The heartbeat is part of the network stack, a protocol called CARP that the OpenBSD developers wrote and FreeBSD has carried in its kernel for twenty-one years, and as it happens the very same interface speaks VRRPv3 as well, so a FreeBSD machine can join a VRRP group on a network where the other routers have never heard of BSD.

Two lines and a vhid

The FreeBSD Handbook sets up a shared address like this, and I have changed nothing but the comments:

# /boot/loader.conf
carp_load="YES"

# /etc/rc.conf
ifconfig_em0_alias0="inet vhid 1 pass testpass alias 192.168.1.50/32"

The first line loads the module at boot. The second gives the network card em0 one more address, 192.168.1.50, in virtual host group 1 (the vhid), guarded by a passphrase that both machines share. The second machine is a doddle: it gets the same line with advskew 100 added, which makes it announce itself a fraction more slowly (by 100/256 of a second, to be exact) and so lose every election it has no business winning.

That is the whole of it, and there is no service to enable afterwards, because nothing runs. The state of the group appears where the address appears, in ifconfig em0, as a line reading carp: MASTER vhid 1 advbase 1 advskew 0 on one machine and BACKUP on the other. Should the network expect VRRP, carpver 3 on the same line switches the protocol, with vrrpprio and vrrpinterval taking over from the skew.

One address shared by two machines clients on the LAN switch host A ■ MASTER em0: own address + 192.168.1.50, vhid 1 advskew 0, virtual MAC 00:00:5e:00:01:01 advertises to 224.0.0.18, IP protocol 112 kernel only, no process host B ■ BACKUP em0: own address vhid 1, advskew 100 listens; takes over after 3 × advbase + advskew/256 of silence, about 3.4 s ~1 per s em1: pfsync, a cable of its own pf's state table travels to the partner as connections open and change

The keepalived equivalent is a vrrp_instance block in a configuration language of its own, followed by systemctl enable --now keepalived, and from that moment the availability of the address depends on a process that the init system has to keep running.

Why OpenBSD built a hovercraft

The reason this lives in a kernel at all is a story the OpenBSD developers tell on the page of their website that collects the songs written for each release. By 2003 they had pf, their packet filter, and pfsync, which copies its state table to a second firewall, and they wanted two firewalls to share an address so that no host on either side would notice one of them failing. The protocol for that was VRRP. Cisco had told the IETF in 1997 that it believed some of its patents covered VRRP, and in March 1998 named the patent on HSRP, its own older protocol; the IETF published VRRP regardless, on terms that allowed patented technology under licence. In August 2002 Cisco's lawyer informed the OpenBSD developers, firmly, that those patents would be defended.

Their answer, in their own words, was to "abandon the wheel entirely and go for a 'hovercraft'". They studied the patent until CARP was "fundamentally different" from it, and they built cryptography into the protocol from the first day. It shipped with OpenBSD 3.5 on 1 May 2004, accompanied by two songs, "CARP License" and "Redundancy must be free". When the developers asked IANA for a protocol number, they were refused for want of a standards body behind them, and so they picked one themselves, a number "which would not conflict with anything else of value". They settled on 112. IANA had given 112 to VRRP, which is why tcpdump(1), left to its own devices, still decodes a CARP packet as VRRP, and why the FreeBSD manual page explains how to stop it with -T carp.

FreeBSD imported carp in 5.4, in 2005. In December 2011 Gleb Smirnoff committed what he called "a major overhaul", released with FreeBSD 10.0, which turned carp from a pseudo-interface of its own into an attribute of an ordinary address; the rc.conf line above has had that form ever since. The patent at the bottom of the whole palaver has long since run its term, and carp now speaks the standard protocol alongside its own.

How the address moves

Once the group is up, the master sends an advertisement to the multicast group 224.0.0.18 every advbase seconds plus advskew 256ths of a second, which with the defaults is roughly once a second, and the backup listens. In CARP mode each advertisement carries an HMAC, keyed with the passphrase, over a counter and every address in the group. With the default multicast, the receiver drops any advertisement that arrives with an IP TTL other than 255, and since every router on the way would have lowered it, an advertisement that passes the check was sent from the segment itself. For networks that dislike multicast, the peer parameter sends the advertisements straight to the partner.

When the advertisements stop, the backup waits three intervals plus its own skew, three and a bit seconds with the defaults, and then takes over. The sequence in carp_master_down_locked() reads like a stage direction: set the state to MASTER, send an advertisement at once, announce the address with a gratuitous ARP (an unsolicited neighbour advertisement for IPv6), start the timers, add the route. The address keeps its MAC as it moves, because carp gives every group a virtual one, 00:00:5e:00:01 followed by the vhid. The neighbours' ARP caches are therefore correct before and after, and the gratuitous ARP is chiefly there to persuade the switch to send the frames down a different cable.

Three intervals of silence, then the address moves −2 s −1 s 0 1 s 2 s 3 s 4 s 5 s A advertises, about once a second A's power supply fails B waits 3 × advbase + advskew/256 = 3.39 s the timer restarts with every advertisement B hears B is MASTER at 3.39 s advertisement, gratuitous ARP, timers, route every 1.39 s

That leaves the machine which is perfectly well whilst the database on it has stopped answering, the case for which keepalived runs track scripts on an interval. On FreeBSD the case is handled by a number in the kernel: net.inet.carp.demotion is added to the skew of every advertisement the machine sends, and the kernel raises it on its own when an interface carrying a vhid goes down or when announcements cannot be sent. Writing to it adds to it, and carp(4) has the purpose in mind: "to control carp behaviour depending on some external conditions, for example on the status of some daemon utility". A health check on FreeBSD is a small loop that runs sysctl net.inet.carp.demotion=240 when the database stops answering and -240 when it comes back. With net.inet.carp.preempt=1 the partner takes the address at the very next advertisement, and devd(8) hands every change of state to whatever script wants to hear about it, through a rule the manual page prints in full.

The same counter keeps a firewall pair in one piece. When one interface of the master fails, the demotion applies to every group on the machine at once, and with preemption switched on the inside and the outside of the network move to the partner together, which is precisely the example carp(4) uses to explain the option.

The state that travels with it

An address that moves is half of a firewall failover. The other half is the state table, since every connection the old master had allowed is unknown to the new one, and a stateful packet filter drops what it does not recognise. pfsync(4) copies pf's states to the partner as they are created and changed, so the backup holds a current copy of the table before it ever needs it. It is a kernel module as well, and it is configured in rc.conf like the address:

pfsync_enable="YES"
pfsync_syncdev="em1"

The two modules know about each other. A machine that has just booted asks its partner for the full table, and if_pfsync.c raises the carp demotion by 240 for as long as the transfer runs. With the default logging the system log records it as it happens, in a line that reads carp: demoted by 240 to 240 (pfsync bulk start) and, once the table has arrived, a second that ends in (pfsync bulk done). The returning machine therefore cannot take the address back with an empty table and cut every connection in passing.

On Linux the counterpart of pfsync is conntrackd from the netfilter project, a separate daemon that keepalived calls through a notify script at each change of state, and the project ships the example script for that purpose.

The same failover, assembled and integrated Debian 13 with keepalived FreeBSD 15.1 with carp HEARTBEAT keepalived, a daemon in userland, kept running by systemd HEARTBEAT carp(4) in the kernel, no process at all CONFIGURATION a vrrp_instance block in keepalived.conf, then systemctl enable CONFIGURATION one ifconfig line in rc.conf, carp_load in loader.conf HEALTH CHECK vrrp_script, a command keepalived runs on an interval HEALTH CHECK a small sh loop writing net.inet.carp.demotion STATE TABLE conntrackd, a second daemon, called through a notify script STATE TABLE pfsync(4) in the kernel, holding carp back until the table has arrived INSTALLED 16 extra packages, 63.6 MiB, Perl among them INSTALLED base system: carp.ko 96 KB, pfsync.ko 101 KB

The sheet

Efficiency. carp.ko is 96,168 bytes and pfsync.ko 101,232 on FreeBSD 15.1-RELEASE for amd64. The protocol is 2,791 lines of C without comments and blank lines, 2,573 of them in the kernel (ip_carp.c and its header) and 218 in the part of ifconfig(8) that configures it. keepalived 2.4.3, counted the same way, has 25,439 lines in its vrrp/ directory, which also holds its configuration parser, its SNMP agent, its D-Bus interface and its firewall integration, and 70,259 lines in all. The binary in Debian 13's package is 914,552 bytes, and installing it on a minimal system pulls in sixteen packages with 63.6 MiB. The timers of carp run as kernel callouts.

What the heartbeat weighs LINES OF C, WITHOUT COMMENTS AND BLANK LINES carp, kernel + ifconfig 2,791 keepalived 2.4.3, vrrp/ 25,439 keepalived 2.4.3, all 70,259 SIZE ON DISK, BYTES carp.ko, FreeBSD 15.1 96,168 pfsync.ko, FreeBSD 15.1 101,232 keepalived, Debian 13 914,552 the vrrp/ directory also holds keepalived's parser, SNMP agent, D-Bus interface and firewall glue

Security. Of the four questions: configuring carp needs root, through ifconfig(8); it takes input from the network by design and accepts an advertisement only with a TTL of 255 and, in CARP mode, a valid HMAC; in CARP mode its parser reads a fixed header of 36 bytes; and a vhid outside 1 to 255 is refused by ifconfig before it reaches the kernel. Of the 727 FreeBSD security advisories published so far, none carries carp or pfsync in its title, the eight on pf mention neither, and NVD lists no entry for either in FreeBSD. The one recent entry that mentions CARP at all, CVE-2026-111217 of 15 October, is a command injection in the web interface of pfSense, where PHP code passed an unchecked vhid to a shell, which places the fault in the page that edits the configuration.

Longevity. CARP since OpenBSD 3.5 on 1 May 2004 and in FreeBSD since 5.4 in 2005, rewritten for FreeBSD 10.0 and unchanged in form since. Maintained in the base system. It lives in the kernels of FreeBSD, OpenBSD, NetBSD and DragonFly BSD; on Linux it exists in userland as ucarp, version 1.5.2 in Debian 13.

Stability. The rc.conf form has been the same since 10.0, and CARP has remained the default protocol with VRRPv3 added beside it, so a configuration written for 10.0 means the same thing on 15.1. The working minimum is two lines, and the failover time with the defaults is three intervals plus the skew, a little over three seconds.

Measured on 16 October 2026: source lines without comments and blank lines in FreeBSD releng/15.1 and keepalived at tag v2.4.3; module sizes with ls -l /boot/kernel on FreeBSD 15.1-RELEASE for amd64; the keepalived binary from the Debian 13 package 1:2.3.3-1; the dependency set resolved from the Debian 13 package index against a minimal system of priority required and important plus openssh-server, sizes from Installed-Size. The failover time is calculated from carp_setrun() in releng/15.1.

The limit

Five limits, and the first is the one every Linux administrator will raise.

The check of the application is yours to write. keepalived brings a scheduler for such checks in vrrp_script, and a set of built-in checkers for the servers behind IPVS. The check itself is a command the administrator supplies on either system, pg_isready perhaps, or a curl against a health endpoint. On FreeBSD the loop that runs it is a few lines of sh(1) kept going by daemon(8) from the base system, and its verdict reaches carp as a single sysctl.

Split brain. If the link between the two machines fails whilst both stay up, each of them declares itself master of the address. Every protocol of this family behaves so, VRRP included, because each machine can only judge by what it hears. The defence lies in the cabling: a second path between the partners, for instance a lagg(4) of two ports underneath the interface.

On a hypervisor the virtual switch has to play along. carp sends from its virtual MAC, and a virtual switch that guards against forged addresses drops those frames until it is told otherwise. Netgate's notes for VMware ESXi list the settings the vSwitch needs, promiscuous mode among them, and say that similar settings exist on the other hypervisors; it is a setting on the port group, made once.

Authentication. In CARP mode the HMAC is built on SHA-1, and the counter it covers is not yet checked against replays; the source says so in a comment of its own, "XXX Replay protection goes here". A replayed advertisement can disturb a group without taking it over, and with multicast only from inside the segment, since the TTL check keeps everything else out. In VRRPv3 mode there is no authentication at all, because RFC 5798 defines none. pfsync carries none either, and its manual page prescribes the remedy: a network of its own for the sync traffic, ideally a crossover cable between the two firewalls, or IPsec. CARP mode with its HMAC is therefore the one to use wherever both partners run a BSD.

The three seconds are calculated. The figure comes from the timers in carp_setrun(), read in the source, and a measured failover wants two machines and a cable to pull.

At three in the morning, when that power supply finally gives up, the kernel on the second machine waits out its three seconds of silence and answers for the address. On FreeBSD the heartbeat lives where the address lives.

Two FreeBSD servers share one IP address with two lines of configuration and no daemon, because CARP is part of the network stack, and the same interface speaks VRRPv3 when the network expects it. pfsync carries the firewall's state table across and holds a rebooted machine back until the table has arrived; on Debian 13, keepalived does the same job as a daemon with sixteen extra packages.