Lean Software ■ The invoice review
The ticket said 14:12. Not on staging, not on the reporter's laptop, not on the machine of the person who wrote the code, and not on mine. It happened on one of four instances, the one that had been up longest, and it involved a page that ought to have been rebuilt an hour earlier and instead served something out of a cache that only that instance knew about. Restarting the fleet made it go away. Nobody could say what it had been.
Ten years ago that ticket would have carried a file name and a line number, and fifteen years ago it certainly would have. The process that produced the fault was the same process you could start on your own machine, with the same code, the same configuration and the same everything, and if it misbehaved twice out of ten times you had a bug you could hunt. The bug lived where you lived.
It has since moved out.
The bug that knows where it lives
The credit side first, because there is one and it matters.
Client-side rendering did genuine harm. A page assembled in the browser shows a crawler an empty document and shows a person on a four-year-old phone a spinner, and for a good decade the industry shipped that to shops and council websites alike. The meta-frameworks answered exactly that, and the people who built them did serious work: streaming responses, and caches that know when a page has gone stale. Anyone who hand-rolled server rendering into a single-page application in 2018 remembers the palaver.
What the credit side does not cover is where the harm came from. The server was removed because it was in the way, and the rendering it had done for free landed in the browser, and once the browser proved a poor place for it the rendering came home. Every layer answers the problem the layer beneath it made, and the problem at the bottom was solved in 1995. So the rendering arrived home with luggage. A build step, a runtime that uses one core per process and therefore scales by multiplication, a hydration pass, and a cache with opinions about time.
That last piece is where the tickets come from. Incremental Static Regeneration works exactly as advertised on one instance, which is how it works on your machine and how it works in the tutorial. Put four instances behind a load balancer and each one keeps its own copy of what is fresh and what is not, so the same URL answers differently depending on which one takes the request. The behaviour is not a defect in anyone's code. It follows from having taken one program and made it into several, whilst the state it depends on stayed in each of them separately.
The technical half of that is the easier half. A fault that cannot be reproduced also cannot be assigned, and the assigning is where the days go.
The ticket is marked ASAP, and it takes four weeks. The alert that starts it off is vague to begin with, which is its own symptom: whoever wrote the threshold cannot know the layer underneath well enough to filter precisely, and by now nobody holds the whole picture. Dynatrace points cleanly at the storage path, and it is not wrong. Storage measure their side correctly, find a question of database logic, and move it to the backend. The backend people are not being awkward when they decline it either: they can show that the sync between node 37 and node 42 has been swallowing writes between half twelve and one for the past fortnight, which is an infrastructure matter and belongs to the platform team. Infrastructure then do the work nobody fancied and find the actual thing. Both environments run their own instance and both mint their own identifiers, so during the shop-data sync one record arrives under two, and the service, handed two identities for one row, aborts exactly as it was designed to.
Nobody made a mistake that has a name. Every department measured correctly on its own side and concluded, correctly, that the fault sat somewhere else, which is this whole argument again with people standing in for services. Four weeks, four teams, a standing meeting to settle who owns the ticket, and a service that behaved correctly throughout. That is the other currency of the coordination rent, and no invoice arrives for it, because it is paid in calendar.
So the environment in which the fault appears is no longer the environment in which the work happens. Development mode differs from a production build, a production build on one box differs from four boxes, and four boxes differ again from the platform where the routing, the cache and the image pipeline are somebody else's implementation. Four environments, one of which you can actually inspect at three in the afternoon, and it is not the one where the customer is.
What the platform layer actually contains
Open source, runs anywhere, no lock-in. Every word of that holds up, and it repays a moment's attention on how little it promises. Self-hosting a modern meta-framework is entirely possible, and the honest way to describe what that takes is to list what OpenNext, the open adapter for other clouds, has to build for you on a public cloud.
- A server-rendering function
- A second function purely for image optimisation
- A third for middleware, because middleware wants to run before routing
- An object store for everything that needs no runtime
- A CDN distribution in front of all of it, with its own compute for request routing
- A key-value store to map cache tags to pages, so that on-demand revalidation can find what to expire
- A queue, so a request that notices staleness can hand the rebuild to something else
- A set of access roles binding these together with least privilege
Eight moving parts, provisioned, monitored and paid for, to deliver the HTML that one process delivered in 1995, still delivered in 2015, and would still deliver today at rather higher throughput. Every one of those parts is defensible on its own. Together they are the shape of a program that was taken apart, and taking it apart is what made the platform necessary.
Nothing in the licence holds you anywhere, and that is the sentence people hear when they hear portable. The estimate for leaving holds you, and the estimate is almost never on the table at the moment a team picks its framework. It arrives two years later, quoted in engineer-months, by which point the argument has stopped being technical. Anyone who watched the container tooling go from free to invoiced has seen the shape of this before.
Which brings us to what the platform sells. Look closely at the layer a hosting company adds on top of a framework that is free and open, and you find coordination. Cache handlers shared across instances so that the four copies agree on what is fresh. Encryption keys that match across a fleet so that a value written on one machine can be read on another. Rollout protection so that a browser holding an older version of your application keeps talking to a server that still understands it, a problem that did not exist when the server sent finished pages and the browser displayed them.
The features themselves run without any of that. The agreement between the pieces does not.
And it is metered accordingly, on the public price list. Twenty dollars per seat per month opens the account, a terabyte of transfer is included and then costs fifteen cents a gigabyte, function invocations start at sixty cents per million, image transformations at five cents per thousand, cache reads at forty cents per million, and cache writes at four dollars per million. Read that last one twice. There is now a price attached to the event of writing something into a cache, an event that in 1995 was called putting a file in a directory and cost nothing at all, because the directory was on a disk in a machine that was yours.
None of this is deception. Every figure sits on a public pricing page and every architectural requirement sits in public documentation. The complexity is not accidental. It is the business model. The mechanism deserves a name all the same, because naming it is what makes it visible: the coordination rent. Take one program apart into parts that cannot see each other, and the agreement between those parts becomes a product with a monthly invoice. Nobody sells you the parts. They sell you the fact that the parts must now agree.
The environment that can reproduce your bug is the one you rent by the month.
The short way back
There is an unglamorous answer to all of this, and it has been sitting in the corner the whole time.
htmx puts reactivity back on the wire: your server sends HTML, the library swaps fragments in response to events, and the state stays in one place because it never left. Version 2.0.10 measures 51,238 bytes minified and 14,996 bytes after Brotli, which is to say roughly fifteen kilobytes for the entire interaction model.
From my own workshop there is something smaller. htm/a takes the same conviction further: the reactivity is fully server-side, the client is a set of custom elements over WebSocket or SSE, and there are no dependencies to audit because there are none at all. Measured this morning the same way, its client comes to 15,286 bytes minified and 4,267 bytes after Brotli. Four and a quarter kilobytes, with a Rust server at the other end holding the state in one process, where it has always been easiest to reason about. It is my own work and it lives on my ecosystem page.
None of this is exotic, and the comparison is not against nothing. Four packages do the job: nginx, PHP-FPM, MariaDB, and the package manager the operating system already ships with. FEMP on FreeBSD or LEMP on Debian, and on the MacBook the same four through brew, one configuration file each and the same versions all the way along. What that buys is the thing this whole piece has been circling. When something breaks in production it breaks on the laptop too, because the laptop is the same stack, and the ticket goes back to carrying a file name and a line number.
Neither of them needs a queue to notice that a page went stale, because a page that is rendered on request is never stale. Neither needs a key-value store to map cache tags, nor protection against a browser and a server disagreeing about which version they are. A single process on a single machine, with all the cores it has, answering requests: FPM has done that quietly for twenty years, and so does any Rust or Go binary you care to compile and drop onto a FreeBSD box with one line in rc.conf.
The point
Back to the question that started this, which is what any of it improved upon a website running on a server you owned.
For a genuine application, where a person stays for hours and interacts hundreds of times, the answer is real and the ceremony pays for itself. For the other ninety per cent, the catalogue, the booking form, the council page, the restaurant menu, the answer is that the browser receives HTML over HTTP, exactly as before, and the machinery to produce it has grown from one process to eight parts and a subscription. That is not progress that was measured. It is progress that was assumed, and then sold back.
The industry has a phrase for the fault you cannot chase, and it used to be an admission of sloppiness: works on my machine. It has quietly become the opposite. The code works on my machine, and it fails on theirs, and theirs is where it lives now.