There is an address that used to hold one of the most argued-over engineering posts of recent years. A team inside Prime Video had built the audio and video quality monitoring the way everyone builds things now, out of small serverless pieces coordinated by a state machine, and it fell over at roughly five per cent of the load it was meant to carry. So they put it back into one process, moved the data through memory instead of through an object store, and the bill dropped by ninety per cent.
Type that address in today and you land on a general Amazon page about entertainment. To be fair, the whole engineering blog under that domain redirects, so this is a site being retired rather than a post being buried. The effect on a reader is the same. The most discussed retreat of the decade is no longer where it was published.
That is the subject of this piece, and the question underneath it is the one this series always asks. Somebody paid for the road out. Who is paying for the road back.
What actually got taken back
Two cases, both public, both with numbers attached.
Prime Video's monitoring service used AWS Step Functions to orchestrate and an S3 bucket to pass video frames between components. Every frame in, every frame out, and the tier-one calls on that bucket did what tier-one calls do. The scaling wall came early and it was hard. The rewrite consolidated the components into a single process on a single machine, passed the frames in memory, and then simply ran several copies of the whole thing where more capacity was needed. Ninety per cent, on their own numbers.
Segment is the better documented of the two, because Alexandra Noonan wrote it up in July 2018 in unusual detail. They had gone from a handful of services to more than a hundred and forty, one per destination, each with its own repository and its own queue. Fifty new destinations meant fifty new repositories. Every worker had to check every queue. A change to a shared library meant a deployment across the estate, and when a test broke somewhere unrelated, somebody still had to fix it before anything could ship.
The numbers after the move back are the ones worth keeping. The test suite went from hours to milliseconds. Improvements to the shared libraries went from thirty-two in a year to forty-six. Nobody got paged any more for a destination processing four events an hour. They also gave things up, and said so: fault isolation got coarser, and the caching got less effective. An honest account, which is rarer than a good outcome.
The house that never went
Now the case that is usually left out, because it does not fit the story about scale requiring sprawl.
Netflix serves its video from Open Connect appliances, and the software on them is FreeBSD. Drew Gallatin's EuroBSDCon talk in 2021 put the figure at four hundred gigabits per second from one machine: an AMD EPYC 7502P, thirty-two cores, two hundred and fifty-six gigabytes of memory, four hundred-gigabit ports. With kernel TLS carrying the encryption, roughly three hundred and seventy-five gigabits came out at about fifty-three per cent processor load.
Four hundred gigabits from one box. Not from a fleet, not from a mesh, from a box, and there is a second one in the rack in case that one dies.
Two things follow from this and both matter. The first is that the same company runs a famously distributed control plane in the cloud, so this is a house that decides per problem where the work belongs. The second is that the work went back into the tree. The sendfile improvements, the kernel TLS, the mbuf changes are in FreeBSD, which means the operator of a two-rack video service gets the same machinery as the operator of a continent's worth of it.
The house that gave it up
WhatsApp is the case that stings, because it is the only house here that moved in the other direction.
Before the acquisition it ran on FreeBSD and Erlang, on hardware the company owned, with something like fifty engineers carrying hundreds of millions of users. The number that circulated at the time was two million connections on a single server, which sounded like a misprint and was not. Then nineteen billion dollars changed hands, and over the following years the service moved onto Linux inside the parent company's container platform.
The reason is the interesting part, and it comes from people who were there rather than from a press release: the move happened because shifting the Erlang code onto Linux was less work than teaching the parent company's operations tooling about FreeBSD. Technical merit never entered the conversation.
What the arrangement achieved afterwards is a matter of guesswork, because nobody has published a comparable figure since. The two million was announced. Whatever replaced it never was.
That is a perfectly rational decision for a firm of that size. It is also the exact shape of everything else in this piece. The efficient arrangement lost on fit, with its merits never once in question.
Who pays for the map
Here is the asymmetry, and it is worth being precise rather than cross about it.
The road out has an industry. Managed control planes, service meshes, tracing vendors, certification programmes, conference tracks, consultancies, a job market that rewards the words on the way in. Every one of those is a business with a marketing budget, and each has a legitimate product: for a house with forty teams shipping independently, that machinery solves a real coordination problem, and the money is well spent.
The road back has nobody. There is no vendor for deleting a queue. No consultancy bills for the removal of a service mesh. No conference track called Fewer Moving Parts, and no certification in Having Left It Alone. The Prime Video team wrote their post and got a fortnight of argument on the internet for it. Segment's write-up remains the reference work eight years on, and it exists because one engineer chose to be candid in public.
So the map everybody navigates by is drawn entirely by people selling the outbound journey. No conspiracy is required to explain it. One direction is a market. The other is somebody with a text editor on a Tuesday afternoon, and nobody has ever sold a ticket to that.
The shorter road
Which brings us to the part that is actually useful, because complaining about maps helps nobody find their way home.
Look again at what Prime Video and Segment ended up with. One process doing the work, several copies of it where more capacity was needed, ordinary boundaries between things that genuinely differ. They arrived there by way of an expensive detour. The arrangement itself is not exotic, and on a FreeBSD machine it is roughly a Wednesday's work.
A jail is a first-class filesystem, so it snapshots and rolls back, and it travels to another machine with zfs send. A thin one weighs a few megabytes until it diverges from its base and starts in a fraction of a second. Nothing supervises it except the kernel, which means there is no management daemon whose crash takes every tenant with it, and no root socket sitting there waiting for its CVE. Ten services on one machine, properly separated, is a configuration file and a few datasets:
# zfs clone tank/jails/base@golden tank/jails/api
# sysrc -f /etc/jail.conf ...
# jail -c apiWhen you need a second machine, you send the dataset over and start it there. That is the same destination the two case studies reached, without the years in between.
And to be entirely clear, because this series is about engineering rather than about taste: none of this rules out the tooling anybody has grown fond of. Grafana runs perfectly well in a jail, as does anything else with a web interface and a fondness for being clicked. The difference is what sits underneath it. A dashboard on a jail is a dashboard you can snapshot before an upgrade, roll back in three seconds when the upgrade was a mistake, and move to a bigger machine by copying it there. Nobody is being asked to give up their graphs, or to enjoy a terminal. The suggestion is only that the floor could be cheaper.
The point
Prime Video went out and came back. Segment went out and came back. Netflix never went and serves four hundred gigabits from a single machine while everyone argues. WhatsApp went the wrong way round, and did it for reasons that had nothing to do with the machines.
The pattern in all four is the same, and it has nothing to do with microservices being wrong. Distribution is a tool for a coordination problem between teams, and it works. It gets reached for as a default by people who have a load problem instead, and the load problem usually fits on one computer, because computers have been getting rather good.
Somewhere on your estate is a service mesh routing traffic between two components that were compiled from the same repository, by the same team, on the same afternoon.