Vivian Voss

Boring Has No Sales Team

it philosophy budgets freebsd accountability

The meeting is in the second week of October, because that is when next year's money gets decided. On one side of the table lies a vendor's deck: forty slides, a dashboard on every one of them, a reference customer somewhere in the middle, and a price on the last. On the other side sits an engineer with a single page. The house runs FreeBSD, and the page says that the logs the vendor proposes to collect are already collected by syslogd, that the metrics already sit in the kernel's own counters, that the firewall on the appliance slide is pf, which arrived with the base system and has been switched on for six years, and that the storage layer the vendor wants to "wrap" is ZFS, which has been snapshotting every dataset in the building since before the vendor existed. The page is correct. Nobody at the table disputes it. The vendor wins.

If you have sat on that side of the table you know what it felt like, and you probably filed it under politics. It belongs somewhere more useful. The people in that room were competent and honest, and the outcome came out of a process doing exactly what it was built to do, which is buying things.

What the process can hear

A budget round is a purchasing instrument. It has a line for the thing to be acquired, a line for what it costs, a signature for whoever approves it, and a report at year end on whether the money went out. Nowhere on that form is there a field for "we already have this". Nobody has ever been asked to justify, in writing, an appliance that was never bought, and nobody has been congratulated for the licence that lapsed because a shell script did the work. An absence does not appear on the form. The engineer's single page describes an absence, and the process has no cell to put it in.

The behaviour this produces can be measured, and it has been. Jeffrey Liebman and Neale Mahoney took the procurement records of the United States federal government and asked what happens when a budget expires at the end of the fiscal year. Spending in the last week runs 4.9 times the weekly average of the rest of the year. The money also gets worse as it gets faster: on a dataset of 130 billion dollars' worth of information technology projects with quality ratings attached, the year-end projects were between 2.2 and 5.6 times more likely to be rated poorly. Use it or lose it is a documented mechanism with a peer-reviewed footnote, and it does not become less real because the house in question is private. A department that returns money is a department that gets less next year, and every manager who has survived two budget cycles knows it.

What an expiring budget does to the last week rest-of-year weekly average = 1.0 1.0 4.9 4.9× in week 52 week 1 fiscal year week 52 Year-end IT projects: 2.2 to 5.6 times more likely to be rated poorly Stylised from Liebman and Mahoney, NBER 19481, AER 2017; the week-by-week series is not reproduced

So the engineer is arguing for a smaller number in a room where a smaller number is a loss. He is proposing that the department demonstrate, for a year, that it needed nothing. That does a career no good at all, and the deck on the other side of the table is the obvious thing to fill the year with.

Who pays for being seen

The deck did not arrive by accident either. In its last full year before Cisco bought it, Splunk reported revenue of 3.65 billion dollars in its 10-K and spent 1.62 billion of that (roughly 1.4 billion €) on sales and marketing. That is forty-four cents of every dollar a customer paid, and it exceeds what the company spent on research and development in the same year by about six hundred million. Datadog, whose dashboards fill most of the slides today, reported 956 million dollars of sales and marketing for 2025 on 3.43 billion of revenue. The numbers are ordinary. Companies that sell software spend on selling it, and the shareholders who read those accounts found nothing in them to complain about.

Where a Splunk dollar went, fiscal 2023 44¢ of every dollar on sales and marketing SALES AND MARKETING 1,621.5 million dollars, 44.4 per cent of revenue RESEARCH AND DEVELOPMENT 997.2 million dollars, 27.3 per cent of revenue Selling the software cost six hundred million more than building it did. Splunk Inc., Form 10-K for the fiscal year ended 31 January 2023; revenue 3,653.7 million dollars

Now the other column. The FreeBSD Foundation, which is the nearest thing the base system has to a voice, published its profit and loss statement for 2025 in June. Total expenses: 2,576,585 dollars and 93 cents (a little over 2.2 million €). About 1.27 million of that went to contractors writing code, and the entire line that could pass for marketing, headed advocacy and education, came to 211,534 dollars. Swag was 6,074.91 dollars, which the accounts record to the cent, and which Splunk's sales operation got through in roughly two minutes of an ordinary working day.

Set the two columns beside each other and the ratio is about six hundred to one on total spending, and something above seven thousand to one on the money spent specifically to be heard. Nobody rigged this. The vendor collects the revenue from the sale, so the vendor pays to be in the room. Nobody collects the revenue from syslogd, so nobody pays to speak for it, and the Foundation's 211 thousand is, in the circumstances, a rather heroic sum. The decider, who has a finite number of hours and a finite number of meetings, hears from whoever paid to be heard. His search costs are real and the deck lowers them. The single page does no such thing, because the engineer who wrote it is, for that afternoon, the whole of the base system's marketing department.

The scale of what the asymmetry moves is on the public record as well. In August 2024 the United States Department of Veterans Affairs signed an enterprise Splunk licence agreement for 118,995,837 dollars and 45 cents over thirty-six months (around 103 million €), through a reseller, on a government-wide procurement vehicle, inside the same federal system whose year-end habits Liebman and Mahoney measured. Whether anyone in that process wrote a single page pointing out that syslog was already there, the record does not say. It would not have had a column for it.

The address for blame

There is a sentence the trade has repeated since the mainframe era, and which IBM, as far as anyone has been able to find, never put in an advertisement: nobody ever got fired for buying IBM. The trade tells it as a joke about cowardice. It is also an accurate description of what a contract is for. A contract is a place that responsibility can be moved to. When the appliance fails at three in the morning there is a support number, a ticket, an account manager, and a firm whose name is on the invoice, and the manager who signed can say, truthfully, that he bought the recommended product from the recognised supplier and that it is now the supplier's problem. Managers have kept their jobs on that sentence, and wanting it is not an unreasonable thing.

Red Hat built an entire company on selling exactly this for software anyone could download, and its own FAQ is admirably plain about the product: tested and certified builds, guidance and stability for "the most-critical environments", support around the clock with no limit on incidents, and, the phrase that matters most, multi-vendor case ownership. Someone will own the case. That is the thing being bought, and it is a real thing.

What the engineer at the table did not know, or did not say, is that the same address exists for the base system he was defending. Klara Systems sells FreeBSD infrastructure support in blocks of between five and fifteen hours a month, with a seventy-two-hour response commitment and an emergency line; the Foundation itself is an address of sorts, and there are others. None of them had a deck in the room, for the reason in the previous section, and so the manager, who was optimising for an address rather than for a stack, had one address on the table and took it. The chair reserved for the base system's supplier stood empty that afternoon, and the manager chose between one address and none.

Two things on the table, one of which nobody sells THE VENDOR THE BASE SYSTEM WHO SPEAKS FOR IT a sales team, 1.62 billion dollars a year WHO SPEAKS FOR IT a foundation, 211 thousand on advocacy WHERE IT APPEARS the deck, the trade fair, the keynote stage WHERE IT APPEARS nowhere: already installed, no field on the form WHO OWNS THE CASE AT 3 A.M. an account manager and round-the-clock support WHO OWNS THE CASE AT 3 A.M. your one engineer, or Klara Systems if anyone asks WHAT THE YEAR-END REPORT SHOWS a line that reads as an achievement WHAT THE YEAR-END REPORT SHOWS nothing, because nothing was bought WHEN THE ENGINEER LEAVES the contract outlives him WHEN THE ENGINEER LEAVES the house carries it, unless it funds a second name The manager is not comparing software. He is comparing the two right-hand columns of blame. Sources in the receipt below

What the decider actually buys

Take the counterargument at full strength, because it is mostly right and the engineer needs to hear it from someone on his side.

The manager looks across the table at the person with the single page and sees, whatever else he sees, a person who will leave. The Bureau of Labor Statistics put the median tenure of an American worker at 3.9 years in January 2024; one analysis of payroll data at the large technology houses put engineers under two, with Facebook near twenty-five months and Uber a little over twenty-one. The knowledge on that page, the six years of pf rules, the zfs send schedule, the jail layout, the reason the syslog filter is written the way it is, lives in one head, and the head has a notice period. The concept has had a name, the bus factor, since 29 June 1994, when Michael McLay asked the Python mailing list what would happen if Guido van Rossum were hit by a bus, and it has had a measurement since 2016, when Guilherme Avelino and colleagues computed the figure for 133 of the most popular projects on GitHub and found that 65 per cent of them would be dead after losing two people. A department runs on the same arithmetic, usually with worse documentation.

Then there is what the engineer himself wants, which the manager has also noticed. A 2021 study presented at ICSE gave the phenomenon its academic name, resume-driven development, and surveyed 591 software professionals about the role of technology in hiring: 82 per cent believed that using fashionable technologies in their daily work makes them more attractive to the next employer, and 60 per cent of the people doing the hiring admitted that fashion shapes the adverts. The engineer with the single page is a statistical anomaly. Most of his colleagues would rather have the vendor's stack on the CV, and the manager, who plans for the colleagues and for the successor and never for the anomaly, is right to notice.

What the manager is buying, then, has very little to do with dashboards. The purchase is a counterparty that will still exist after the engineer has gone, and a case that somebody else owns. On top of that comes a line in the year-end report that reads as an achievement, a number that does not shrink next year, a decision his own superior has heard of, and a logo the auditors will not query. Underneath all of it sits insurance against the day the one person who understood the base system hands in his badge, and the premium gets paid with somebody else's stack.

For a house that plans its engineers as replaceable, every one of those decisions is rational.

The limit

The position has edges, and they should be stated by someone who holds it.

"We already have this" is a promise. It promises that a named person will be awake at three in the morning, will know why the filter is written that way, will still be employed in eighteen months, and will pick up when the number on the screen is the chief executive's, and the house has to believe all four before the single page is worth anything. On 24 March 2022 Crossref, the organisation behind the DOI infrastructure that much of academic publishing depends on, went dark for seventeen hours over an incomplete BGP configuration on the provider side. Geoffrey Bilder's post-mortem is honest to the point of discomfort: the infrastructure group was two people short, one of them having recently left for a startup, and "our one long-suffering sysadmin had to field this all by himself." Read as an argument, the outage points at the second person and nowhere else. A house that runs on what it already has must plan for that person, and a house that will not fund the second person has, in effect, already decided to buy.

The base system also arrives with no warranty, in capital letters, and nobody may demand one from a volunteer. The address for blame that Klara provides is bought, like Red Hat's, and a house that wants it has to pay for it and, before that, has to know that it exists. Sometimes, too, the single page is simply wrong. The base system logs one machine superbly and a fleet of four hundred rather less so, and there are people in every house who need a picture rather than a shell, and their needs are no character flaw. And the mechanism described here does not dissolve on being understood. It changes only when the house changes what it is planning for.

The other piece of paper

The engineer's mistake, and it is a generous one, was to argue about technology in a room that was pricing accountability. The page said that the base system did the job. The room needed to know who would stand behind it. Those are different questions, and the second one has answers that fit on the same page: a support quote from a firm that will own the case; a second name, trained, with a date by which the training is done; the Foundation's number, which looks like a rounding error next to the vendor's marketing and does the same work for a fraction of it; and the vendor's own sales-and-marketing line, so that the room knows what it has been paying to hear. There is one larger answer as well, which is the house deciding that its engineers are the thing it intends to keep.

Netflix made that decision. It kept the people who understood the base system, paid them to understand it better, and the result was a single FreeBSD machine serving four hundred gigabits of video a second, with the work handed back into the tree where the next house can find it. That room had no vendor in it either. The company had chosen to be its own address for blame, and found, as Chapter 12 of Integrated by Design argues, that responsibility in a system built by one team is a chain with an owner at every link.

Next October the deck will be back, forty slides, the same dashboards. Across the table there will be a page again. This time it should say who will be there at three in the morning, in two names, with a phone number under each.