A production system is behaving oddly. Not broken, just odd: response times have grown a second hump, and nobody knows what from.
What happens next? In most shops somebody opens a web interface, hunts for the right time window, lays two curves on top of one another and begins to guess. If the quantity they want is not among them, a ticket gets written so that it will be emitted from the next deployment onwards. An answer in a fortnight, with luck.
On that same machine sits a tool that could settle the question in forty seconds. It is installed. It costs nothing. It does not ask whether anybody instrumented a field in advance.
This piece is about why nobody reaches for it.
What Was Written Down in 2004
In 2001 three kernel engineers at Sun Microsystems began work on a problem as old as running computers for a living. You can measure a system while it sits in a lab, and you can watch it while it is small. Once it is large and customers are on it, both of those stop.
Bryan Cantrill, Michael Shapiro and Adam Leventhal took twenty-three months to the first integration, which puts it at 2003. It shipped with Solaris 10 in March 2005. A year before that they presented their paper at USENIX, under the title Dynamic Instrumentation of Production Systems, and its abstract reads like a specification somebody actually met.
Three sentences from it repay a careful reading.
The first: DTrace can instrument both user-level and kernel-level software "in a unified and absolutely safe fashion". Absolutely safe. Not handle with care, not recommended on test systems.
The second: "When not explicitly enabled, DTrace has zero probe effect", and the paper spells out what that means, namely that the system operates exactly as if DTrace were not present at all. Switched off, it is gone. Not one per cent, not half of one, nothing.
The third concerns quantity. The facility allows for many tens of thousands of instrumentation points, and even the smallest of systems offers "on the order of 30,000 such points in the kernel alone".
And in the same passage the authors say why they held themselves to those two properties: "To be acceptable for use on production systems, performance analysis infrastructure must have zero probe effect when disabled, and must be absolutely safe when enabled."
Cantrill later put it more casually, and in that form it sticks: if you run a tool and the system dies as a result, you will never be allowed to run that tool again.
Not a Feature, a Decision About Where Things Live
You can read that abstract as marketing copy. It is an architectural commitment, and the difference shows in where the thing belongs. What they built is dynamic instrumentation, which is a capability rather than a product, and a capability has to live somewhere.
An observability tool that gets installed afterwards is always one step late. It is not there when you need it; you procure it once something has happened. It has requirements of its environment. It goes through an approval. It costs money the moment it runs, and so it runs only where somebody managed to justify the cost.
A tool that costs nothing while switched off can sit everywhere. That was precisely the intention. DTrace has been part of the base system since Solaris 10, and wherever it travelled that property travelled with it. FreeBSD took it in. macOS has it. In neither case is it a product you buy; it is something you type.
Which is the Unix way of building things, demonstrated in a single example. Not one large program that answers everything, but a capability in the system from which you assemble the question yourself.
The Other Answer
Linux went another way, and telling that story fairly means going back further than 2014, and starting with why it had to go another way at all.
DTrace was published under the CDDL. The Linux kernel is GPLv2. The two licences do not sit together, and no amount of goodwill on either side changes that; a CDDL-licensed tracing framework cannot simply be merged into a GPLv2 kernel. Oracle released the DTrace kernel code under GPLv2 and later in August 2017, by which time the question had long since been answered differently.
The Berkeley Packet Filter dates from 1992, from Steven McCanne and Van Jacobson, and was originally exactly what the name says: a small virtual machine in the kernel deciding which network packets travel upwards. A frugal, handsome thing with a narrow purpose.
In 2014 Alexei Starovoitov and Daniel Borkmann took that mechanism and made something considerably larger of it. Ten registers, a 512-byte stack, a JIT compiler, and above all a verifier. It went into Linux 3.18.
That verifier is where the two answer the same question by opposite means, and the distinction is worth stating precisely. DTrace constrains what can be written: its language has no loops and no arbitrary control flow, so a dangerous script is not expressible in the first place. eBPF admits arbitrary programs and then proves them safe, with static analysis at load time that walks every possible path, forbidding loops, invalid jumps, invalid memory access and the leaking of kernel data.
Those are not the same guarantee. One is enforced by construction, the other by an analyser that grows with every new capability, and an analyser is software like any other. That difference has a record, and it is not a quiet one, but it belongs to a different piece than this.
On top of that sits bpftrace today, written by Alastair Robertson with Brendan Gregg among its principal contributors, and deliberately shaped so that somebody with DTrace in their fingers finds their way about at once.
Which matters for the bill further down: both sides can trace a live production system without stopping it. On capability there is nothing to choose between them.
What Happens When Somebody Measures Both
The temptation at this point is to look for a winner, and there is even a paper that would seem to offer one if you read it in a hurry.
Mateusz Piotrowski presented work at AsiaBSDCon in 2024, published in the conference proceedings, measuring the overhead of DTrace on FreeBSD against eBPF on Linux, with a microbenchmark and an application benchmark built on the experiments from Gregg's book.
And then he writes the sentence that keeps this whole piece honest: it is not possible to say which system is faster. There are too many factors he did not account for, and any performance comparison between DTrace and bpftrace "would be unfair and misinformed".
His conclusion is equally unexcitable. The two are entirely different pieces of software, even though they might look similar.
You can pour a great deal of work into a study and end up establishing that the question you started with was the wrong one. That is not a failure. That is a result.
One aside from his future-work section deserves a place all the same, because it concerns operations rather than arithmetic. During early experimentation bpftrace took roughly ten minutes to print its collected statistics and quit, where DTrace, when signalled, was done in a few seconds. He records it as an open question for the bpftrace community rather than as a verdict, and that is how I pass it on.
The Bill
So the Unix line has a tool in the core of the system that can answer any question at all about a running machine, immediately, locally, with no preparation and no running cost, and Linux has built itself an equivalent.
And what do people reach for?
Infrastructure monitoring starts at fifteen dollars per host per month. Application performance monitoring at thirty-one. Logs cost ten cents per ingested gigabyte, and are then charged a second time at one dollar seventy per million indexed events. Multiply that by a fleet and a generous logging policy and you get an invoice nobody can estimate in their head any more.
How large that can grow is now a matter of record. On a Datadog earnings call on 4 May 2023, an analyst worked out from a remark by the finance chief about a large non-recurring bill that the figure must be about sixty-five million dollars. Gergely Orosz went after it and had it confirmed by current and former engineers: Coinbase, for the year 2021.
That is an extreme from the height of the crypto boom, and I do not raise it to sit in judgement on Coinbase. I raise it because it demonstrates that there is no ceiling in that direction. Upwards, the thing is open.
The everyday finding is the smaller one. When a service misbehaves, the first question in most shops is not "what is this process actually doing" but "have we got that in the dashboard". And if it is not in the dashboard the question is treated as unanswerable, while on every one of those machines sits a tool built for exactly that purpose, by people who sat down twenty-two years earlier and wrote out which guarantees it would have to keep.
The Limit
Now the honest part, because the opposing case is strong.
A local tool answers local questions. It will tell you everything about this one machine and nothing whatsoever about the four hundred beside it. Aggregating across a fleet, retaining months of history, following one request through seven services: those are real problems, and DTrace solves not a single one of them. It was never built to.
The timing is just as real. Somebody woken at three in the morning wants to see a curve, not write a script. The platforms are not merely selling storage, they are selling readiness: somebody decided in advance what would be measured, so the answer is already sitting there when the question arrives. That is worth money, and anybody who disputes it has never carried a pager.
And these tools are not free merely because they cost nothing. Asking a running system a sensible question requires knowing what you are looking for. That is a skill, it needs practice, and it decays when nobody has occasion to use it.
Nor is the DTrace side without a loaded weapon of its own. dtrace -w is a destructive mode that can modify a running kernel. It is separately gated and requires a deliberate decision, but it exists, and an operator who grants it has granted something serious.
So the complaint is not with the platform. It is with the order of operations.
The Point
In 2004 somebody wrote down the two properties an observability tool must have before you are entitled to switch it on when it matters: nothing at all when disabled, absolute safety when enabled. Both sides of that divide went after those properties, by opposite means, over twenty years and with a licence quarrel in the middle.
The result is sitting on every machine you run.
When the next service misbehaves, the first question is not whether the quantity is in the dashboard. The first question is whether anybody in the room can drive the tool that is already there. If somebody can, the answer costs forty seconds. If nobody can, it costs a fortnight and a line item on an invoice with no known ceiling.