Vivian Voss

Read on LinkedIn

The Log

unix freebsd linux devops

The Unix Way ■ Episode 9

Eric Allman wrote syslog in the 1980s for Sendmail. It became part of BSD. Then part of every Unix system. RFC 3164 documented it in 2001. RFC 5424 standardised it in 2009. Forty years of production. Zero licence fees.

Yesterday's observability post drew comments insisting that tail cannot replace a monitoring stack. Quite right. Nobody suggested it should. But one would rather like to demonstrate what the built-in tools actually do, before dismissing them over a dashboard.

FreeBSD

Log rotation. One line in /etc/newsyslog.conf, executed hourly by cron:

/var/log/app.log  644  7  1000  *  JC

Seven compressed archives, rotated at 1 MB. No Loki. No Fluentd. No YAML. The J flag compresses with bzip2. The C flag creates the file if absent. newsyslog has done this since 4.4BSD. One rather appreciates a tool that has not needed a rewrite in three decades.

One Line. Complete Log Rotation. /var/log/app.log file path 644 permissions 7 archives 1000 KB limit * any hour J bzip2 C create newsyslog: since 4.4BSD. No YAML. No vendor. No rewrite in three decades. Compare: a typical logrotate config is 8 lines for the same result.

For comparison, the same task in Linux's logrotate requires a separate configuration block:

/var/log/app.log {
    rotate 7
    size 1M
    compress
    missingok
    create 644 root root
}

Seven lines for what FreeBSD does in one. Both work. One is a sentence. The other is a paragraph.

Live error monitoring across every log on the system:

tail -f /var/log/*.log | grep -i error

One pipe. Every log. Every error. In real time. The glob expands to every log file. tail -f follows them all. grep selects what matters. The output is immediate, human-readable, and costs precisely nothing.

The five noisiest processes today:

awk '{print $5}' /var/log/messages \
  | sort | uniq -c | sort -rn | head -5

Four commands, piped. awk extracts the process field. sort groups identical entries. uniq -c counts them. sort -rn ranks by frequency. head -5 shows the top five. This is frequency analysis. A data scientist would recognise the pattern. A vendor would charge for the dashboard.

Notification when something critical appears:

tail -f /var/log/app.log \
  | grep --line-buffered "FATAL" \
  | while read line; do
      echo "$line" | mail -s "FATAL" ops@example.com
    done

tail follows. grep filters. while read iterates. mail notifies. That is PagerDuty in four lines. It runs in a tmux session, costs nothing, and has no vendor who can revoke your access overnight.

Four Patterns. Four Pipes. Zero Vendors. Live Monitoring tail -f | grep Every log. Every error Real time Frequency Analysis awk | sort | uniq -c Top offenders Data science in a pipe Alerting grep | while read | mail PagerDuty in 4 lines No vendor lock-in Log Rotation newsyslog.conf One line Since 4.4BSD | | | Each command does one thing. Together, they do what a vendor contract promises. Log rotation + live monitoring + frequency analysis + alerting. Four commands. Each composable. Each free. Each older than most frameworks.

Linux

journalctl, built into every systemd installation since 2012, provides structured logging with native filtering:

journalctl -f -p err

Live errors. One command. The -f follows, the -p err filters by priority. No grep required, because the structure is built in.

journalctl --since "1 hour ago" -p err | wc -l

How many errors in the last hour? Bounded time queries, natively. No Prometheus range selector. No PromQL. A human sentence as a time specification, which is rather more civilised than time() - 3600.

journalctl -o json \
  | jq -r '.SYSLOG_IDENTIFIER' \
  | sort | uniq -c | sort -rn | head -5

Structured extraction via JSON and jq. The same pipe pattern as FreeBSD, but feeding from structured data rather than parsing text. Both work. Both compose with the same pipes. Both cost nothing.

Two Systems. Same Pipes. Same Result. FreeBSD newsyslog.conf: 1 line syslog: plain text, grep-friendly tail -f | grep | awk | mail Since 4.4BSD. Never rewritten Linux logrotate: 7 lines journald: structured, JSON-native journalctl -o json | jq | sort Since systemd 2012. Binary format Both compose via pipes. Both cost nothing.

The Proof at Scale

Netflix serves a third of global internet traffic from FreeBSD-based Open Connect Appliances. Their content delivery network, the largest single consumer of bandwidth on Earth, runs on an operating system that ships with newsyslog and syslog in the base system.

WhatsApp handled 900 million users with 50 engineers on Erlang and FreeBSD. Fifty engineers. Nine hundred million users. Eighteen million users per engineer. Neither Netflix nor WhatsApp required Grafana to read their logs.

The tools compose: tail filters, grep selects, awk aggregates, mail notifies. Each does one thing. Together, they do what a vendor contract promises and rather frequently fails to deliver.

The Curriculum Question

Every command in this article has existed for decades. None requires installation. None requires configuration beyond a single line. None requires a subscription, a deployment pipeline, or a Kubernetes cluster to run a sidecar container that watches your logs being watched.

And yet, ask a junior developer to find the five noisiest processes on their server and watch the silence. Not because the task is hard. Because nobody taught them. Bootcamps teach Grafana. Universities teach vendor stacks. The man page sits unread, patiently waiting, the way it has waited since the 1980s.

The difference is not capability. It is curriculum.

Log rotation, live monitoring, frequency analysis, and alerting. Four commands. Each composable. Each free. Each older than most of the frameworks they are supposed to be replaced by. The tools were always there. The curriculum was not.