Grafana, or Two Commands

At twenty past four in the morning a mail arrives. Subject line from a machine, body from periodic, and somewhere in the second paragraph sits the sentence that explains the whole night: a filesystem crossed eighty per cent, and here are the twenty lines the system logged while it happened. Whoever reads that mail learns what went wrong, in the words of the program that had the trouble, at the moment it had it.

That is the difference this piece is about, and it is a difference of address. A watcher that lives inside the system reports the symptom and the reason in one breath, because both were written a few lines apart in the same file. A watcher standing outside reports the symptom, and then somebody has to go and fetch the reason.

Everything else follows from that, including the bill.

What the scale says

The measurement ran in a FreeBSD 15.0-RELEASE jail on my own hardware, rolled back to a golden snapshot so nothing from previous work could contaminate it. Host load at the start was 0.10, which is as close to nothing as a running machine gets.

The jail before any of this held 397,508 kilobytes on disk, eight processes, one package and fourteen base-system tools that already did the job, the package in question being pkg itself. Those fourteen are worth naming, because the whole argument turns on them:

syslogd  newsyslog  periodic  accton  sa  lastcomm
vmstat  iostat  top  systat  dmesg  last  cron  mail

Alerting, event history, daily reporting, process accounting and live inspection, all present, all documented in the manual, all costing nothing. Then three commands, pkg install prometheus grafana node_exporter, and the arithmetic afterwards.

Disk went to 1,085,564 kB, a difference of 688 megabytes. Starting the services once pushed it to 1,351,227 kB, so the first start alone laid down another 259 megabytes before a human had looked at anything at all. Resident memory settled at 470,104 kB three quarters of a minute later, of which Grafana holds 373,664, Prometheus 80,396 and the node exporter a modest 16,044. Eight processes had become fourteen.

Two things in fairness: the package metadata claims 1,130,627,388 bytes and ZFS compression brings the actual footprint down to 688 megabytes, so both figures are true and I quote both. And the installation pulled in no dependencies at all, because Go binaries are statically linked. That is genuinely tidy, and rather better than most stacks manage.

Now the number that made me stop and check it twice. Grafana's own database, /var/db/grafana, before a single dashboard exists and before anybody has logged in: 265,768 kilobytes. Two hundred and seventy-two megabytes of nothing in particular.

The entire log directory of that jail, holding every event the operating system has recorded since it booted, is 79 kilobytes.

Two commands, one minute

Numbers about storage are one thing. The question that matters on a bad morning is how fast somebody gets to the cause, so I staged an incident and put a clock on it. A process went into a tight loop and pinned a core, which is about as ordinary as failures get.

top -b -o cpu put the offender at the head of the list two milliseconds after I asked. Then ps -axo pid,pcpu,command | sort -rnk2 printed this:

30881 99.1 sh -c while :; do :; done

Three milliseconds of machine time, and that line carries the whole command, the process identifier and the user who owns it. procstat added the binary path a millisecond later, in case anybody wanted to know which sh it was.

Those milliseconds are the honest measurement and the dishonest headline, so let me put the human number next to them. Opening a terminal, typing two commands and reading two answers takes about a minute, and that minute is the figure worth comparing with anything. Two commands, both in the base system since before some of my colleagues were born, one minute from wondering to knowing.

At the same second the exporter was serving its metrics, and here is what it had to say about the same event:

node_cpu_seconds_total{cpu="0",mode="user"} 267.17
node_cpu_seconds_total{cpu="1",mode="user"} 314.09

Two hundred and six metrics, one per counter per core, and not one of them contains the string sh -c while. That is not a flaw in the exporter, which does exactly what it was built to do: it reports the state of the host. Nothing in that counter was ever going to name the process. The dashboard will draw a beautiful step change at 05:52 and it will be entirely correct and it will not tell you who did it.

So the honest word for the second path is later: see the step, log in, find the window, pick the host, open the log. Anybody without shell access on that host stops at step two and writes a ticket, and this is where the minutes stop mattering altogether.

That ticket, in my experience, has something of a career ahead of it. It waits for the morning review, where it is discussed by people who were not there when it happened, and assigned to somebody else who also was not. It comes back with a question. It goes out again. Rather often it is closed a fortnight later with a note observing that the thing has not recurred, which is true and explains nothing. One minute at the terminal, then eleven working days at the process, and the eleven days are not the tool's fault in any technical sense: the tool simply put the answer somewhere that required a meeting to reach.

Unless the fault is stopping the business, of course. Then the whole apparatus is quietly stepped around, somebody with a key logs into the box, and runs top.

The watcher nobody watches

Here is the part that gets left out of every comparison I have read, and it is the one with real money attached.

Prometheus and Grafana do not run in the air. They need a machine. That machine needs patching, needs backing up, needs a certificate before somebody puts a login screen on the open internet, which is a small kerfuffle of its own, and it needs monitoring, because a monitoring server that has quietly fallen over does not send an alert saying that it has quietly fallen over. The failure of the alarm is the one failure the alarm cannot report.

Four hundred and seventy megabytes of resident memory and a gigabyte of disk amounts to an instance in its own right, and instances have invoices. Then somebody has to learn the query language, and in larger houses that somebody becomes a team, and that team sits between the systems and the people who run them, and reports to them.

The base system tools have none of that recursion. top runs on the machine you are already worried about. If that machine is dead, you know, because it is dead.

The language you cannot take with you

The remaining case for the stack, and it is a real one, is the query language. Ninety-fifth percentile of response time across fifty hosts, a rate over a five-minute window, a sum across a fleet: that is genuinely awkward with awk, and PromQL does it in a line.

Two things are worth saying about that line, though, and neither appears in the vendor material.

The first is that the skill does not travel. Somebody who knows grep and awk has known them for thirty years and will use them on any machine they ever sit at. Somebody who knows PromQL knows Prometheus. The moment the house changes tooling, the second person starts again, and the first person does not notice the change. That is a commercial property wearing technical clothes, and it does rather well at what it looks like it is doing.

The second is that a query language is a new place for bugs to live. A wrong grep returns nothing and you notice within the second. A wrong PromQL expression returns a number, and the number looks fine, and the alert built on it fires late or never or constantly. Alert fatigue is usually described as a human problem. A fair part of it is a syntax problem in a language most of the people on call do not write daily.

Where it earns its keep

Now the other side, in its strongest form, because this is a measurement and not a sermon.

Fine resolution is real. Fifteen-second sampling catches a spike that a daily report averages into invisibility, and if the question is whether latency degraded gradually over three weeks, no log will answer it. Aggregation across a fleet is real: nobody greps fifty hosts by hand at four in the morning. And pictures for people without shell access are real, because a product owner asking whether Tuesday was worse than Monday deserves an answer that is not a terminal.

But look at what those three cost when you buy them separately. Central syslog is in the base system: one line on the clients, one flag on the collector, and the fleet's logs sit in one place where grep behaves exactly as it always did. For the pictures, munin-master and munin-node together weigh 2,533,160 bytes and draw graphs on a web page all day long. That is a factor of 446 against the stack measured above, for the specific job that Grafana is usually wheeled in to do.

The gigabyte does not buy visibility. Visibility was already there, and part of it was better. The gigabyte buys resolution and a query language, and those two things should be bought deliberately, by somebody who has said out loud what they are for.

The point

The claim under examination was that without this stack you are flying blind. On the machine in front of me, before any package was installed, syslogd was writing events with timestamps, newsyslog was rotating them, periodic was preparing a daily report to mail out, and accton was ready to account for every process that ran. Fourteen tools out of fourteen, present, documented, and costing nothing.

None of this means the stack is wrong. It means the sentence used to justify it is, and a different sentence would have to be defended out loud: that fifteen-second resolution across fifty machines is worth a gigabyte, a second machine, a specialist and the training that keeps them. Some houses should. Most of the ones I have walked into were paying for the resolution and using it to answer questions their logs had already answered.

The same applies on Linux, incidentally, where all of this runs on the shell just as well and the tools merely have different names.

That mail at twenty past four came from the machine that had the problem, and it brought the reason with it, because it was standing close enough to see.