The Unix Way ◆ Episode 16
On a busy Linux load balancer one types netstat
-anp. The prompt is in no particular hurry to come
back. Time enough, on a generous estimate, to make a cup of
tea and consider one's life choices. On a FreeBSD host
doing roughly the same work, the same command returns
before the kettle has finished. Both tools speak Unix text.
Both pipe into grep, awk and
sort without fuss. Only one of them was built
the way Unix tools are built: ask the kernel, format the
answer, get out of the way.
This is not a story about Linux being slow. It is a story about where the Unix text interface lives, and what that choice costs a decade later.
The Principle
Doug McIlroy's
principle, the one that gets quoted at every Unix lecture,
says: text streams are the universal interface. The Unix
Way is to build tools that answer questions in text and
let pipelines compose them on the fly. The interface lives
in the tool's mouth. One asks netstat; it
speaks. One pipes its output into grep; that
speaks. The text exists where one program hands its work
to another or to the human at the keyboard, and not
before.
FreeBSD took the principle and built netstat
accordingly. The tool asks the kernel through
sysctl, formats one answer at its output,
pipes onward. Keep It Simple, Stupid in the
classical sense. Linux took the same principle and added
something McIlroy never asked for: a file shaped
like a text interface that isn't quite one.
/proc/net/tcp is a kernel-rendered ASCII
dump. It is readable, in the sense that one can
cat it. It is not askable: there is no way to
query it for "sockets on port 443" or "all UDP listeners".
One can only read it whole. So the tool behind it still
does all the work, plus the parsing of someone else's text
on the way in.
A Short History of netstat
netstat was part of the
4.2BSD networking release of August 1983.
The CSRG team at Berkeley, working with the new TCP/IP
stack that Bill Joy
and Sam Leffler had brought together, needed a tool that
could ask the kernel about its sockets, its routing table,
its interfaces and its protocol statistics. The result was
netstat(1). The vocabulary it established
(-a for all sockets, -n for
numeric output, -r for the routing table,
-i for interfaces, -s for
protocol statistics) became part of the shared idiom of
every Unix administrator for the next several decades.
For most of its history, netstat was a thin
Unix tool: ask the kernel, format the answer, send it to
standard output. On BSD it asked through
sysctl(3) and kvm(3): interfaces
designed to be queried by a tool. On early Linux it asked
through /proc/net: files designed to be read
whole, in ASCII, by humans and tools alike. The format on
the screen looked the same. The shape of the interface
underneath did not. The first is a tool asking another
tool a question. The second is a tool reading a file that
someone else has already laid out, and which one cannot,
in any meaningful sense, ask.
FreeBSD: One Tool, Still in Base
On FreeBSD in 2026, the answer to "how do I list all the established TCP connections on this host" is the same as the answer in 1996, in 2006, and in 2016:
netstat -an
To get a specific protocol family:
netstat -anp tcp
netstat -anp udp
netstat -an -f unix
For the routing table, the same convention:
netstat -rn
For protocol statistics, the same again:
netstat -s
netstat -s -p tcp
The tool is part of the base system. The man page lives at
/usr/share/man/man1/netstat.1.gz. The source
lives under /usr/src/usr.bin/netstat/. It is
updated as part of the FreeBSD release cycle, by the
people who also maintain the kernel that produces the
data.
The reason it is still fast on a busy server is the
interface underneath. The FreeBSD kernel exposes its
in-kernel
protocol control blocks
through the sysctl tree:
net.inet.tcp.pcblist(TCP PCBs)net.inet.udp.pcblist(UDP PCBs)net.inet.raw.pcblist(raw sockets)net.local.stream.pcblist(Unix-domain stream)net.local.dgram.pcblist(Unix-domain datagram)net.local.seqpacket.pcblist(Unix-domain seqpacket)
A single sysctl call returns a structured
array of records. The userland netstat
formats them into columns, applies the requested filters,
and exits. The interface here is netstat
itself: one asks netstat, netstat
speaks Unix text back, and that text pipes into
grep, awk, sort or
any other tool one cares to compose it with. There is no
intermediate file pretending to be a tool. The cost is
linear in the number of sockets, with a small constant; on
a server with sixty thousand connections the command still
returns in well under a second.
For users who prefer a more focused view of sockets without
the routing-and-statistics surface area of
netstat, FreeBSD also ships
sockstat(1) in base:
sockstat -4l # IPv4 listeners
sockstat -p 443 # everything on port 443
sockstat reads the same sysctl
tree. It is a complementary tool, not a replacement, and
it has been in the base system since FreeBSD 3.0 in 1998.
Linux: netstat, Still There, No Longer Maintained
On a Linux system, netstat is also still
there, in most cases. It lives in the
net-tools package, alongside
ifconfig, arp, route
and a handful of other tools that the BSD-trained
administrator has known since the late 1980s. The Linux
versions were written in the early 1990s by Fred N. van
Kempen and others, modelled on the BSD originals but
implemented from scratch against the Linux
/proc filesystem rather than against
sysctl and kvm.
For most of Linux's history, this worked.
/proc/net/tcp contains one line per active
TCP socket, in a well-defined ASCII format. The Linux
netstat opens that file, reads the lines,
parses each one, looks up process information through
/proc/<pid>/fd/, and prints the result.
On a small or medium server with a few hundred sockets,
the cost was invisible.
The deeper issue is the shape of the interaction, not the
byte count. /proc/net/tcp is a file. One can
read it. One cannot ask it anything. If one wants only
sockets on port 443, one has to read the whole file and
filter in userspace. If one wants only established
sockets, same thing. If one wants only sockets owned by a
particular process, one has to read
/proc/net/tcp whole and walk
/proc/<pid>/fd/ whole, matching inodes
by hand. The Unix Way says "compose tools by piping text";
/proc/net/tcp is not a tool, it is the text
pretending to be one, and the actual work still happens in
the tool behind it.
On a busy server the pretence becomes expensive. A reverse
proxy or a load balancer in 2010 might have sixty thousand
simultaneous connections, perhaps far more. Listing them
with netstat -anp meant reading sixty
thousand lines from /proc/net/tcp, parsing
them all, walking the /proc/<pid>/fd/
tree, and filtering in userspace. On the same server,
netstat could take ten or fifteen seconds and
burn rather a lot of CPU during that time. None of that
cost was paid by the kernel; all of it was paid by the
tool, because the file in front of the tool was not
askable.
The net-tools package itself has not had a new
release since 2011. The maintainer publicly attempted to
deprecate it in 2009; the LWN article
"Moving on from net-tools"
documents the conversation. Most major distributions now
recommend iproute2 as the replacement, and
several no longer install net-tools by
default. The Linux netstat is still there, it
still works for small workloads, and it still does the
wrong thing on busy ones. The wrong thing is not the
tool's parsing; it is being asked to do that parsing
because the thing in front of it was a file rather than a
queryable interface.
Linux: ss, the Unix Pattern Restored
The replacement is ss(8), short for
socket statistics. It was written by Alexey
Kuznetsov, the same author who wrote much of the Linux QoS
code and started the iproute2 package.
ss does not read /proc/net/tcp.
It asks netlink instead, and
netlink, crucially, is askable.
netlink
is a Linux kernel-to-userspace IPC mechanism, also
Kuznetsov's work, designed for structured queries against
kernel subsystems. In October 2005, Linux 2.6.14 added a
netlink family called NETLINK_INET_DIAG,
exposing IPv4 and IPv6 socket information as a queryable
interface. In March 2012, Linux 3.3 generalised it to
NETLINK_SOCK_DIAG,
adding Unix-domain socket support. The userland tool opens
one netlink socket, sends one request specifying exactly
what it wants, and receives back only the matching sockets
as a stream of inet_diag_msg or
unix_diag_msg records. ss
formats those records into Unix text, prints them, and
exits. The interface here is ss itself: one
asks ss, ss speaks, and the
answer pipes onward like any other Unix tool's output.
The performance difference is not a constant factor. On a server with a few hundred sockets it is barely measurable; on a server with sixty thousand it is the difference between a snappy tool and a slow one. The structured query also allows filtering in the kernel rather than in userspace:
ss -tan state established '( dport = :443 )'
The bracketed expression is translated into a kernel-side
filter, and only the matching records come back. Both
gains follow from the same source: the kernel offers a
queryable interface, and ss is free to be the
Unix text tool one talks to. The middleman that pretended
to be one is gone.
In day-to-day use:
ss -tan # all TCP, numeric
ss -tanp # add process info (needs root)
ss -s # summary by protocol
ss -tan state established # only established TCP
ss -lntp # listening TCP with processes
The vocabulary is similar enough to netstat
that a long-time BSD administrator can guess most of it on
first encounter. The behaviour is fast enough that it
survives in production.
The Architectural Point
The Unix Way puts the text interface in the tool's mouth.
netstat IS the interface: one asks it, it
speaks, the answer pipes into grep,
awk, sort, less and
whatever else one cares to compose. That is McIlroy's
text-streams principle in its proper place. KISS in the
classical sense: one tool, one question, one answer.
/proc/net/tcp is a different idea altogether.
It is a file shaped like text. One can cat
it, one can squint at it, one can grep it in
a pinch. It is not a tool one can ask: there is no syntax
for "established sockets on port 443 owned by nginx". So
the tool behind it has to do that work: reading the whole
file, parsing every line, filtering in userspace, walking
another /proc tree, and only then formatting
the result for the human. The Unix-ness has been pushed
one layer out, and the work in front of it has not gone
anywhere.
This is not a criticism of /proc as a debugging
surface. Being able to grep /proc/net/tcp for
one weird socket without writing a tool is a real benefit,
and it is the reason /proc is still there in
2026. The trouble is that the same surface was also asked
to serve as the production data feed, and a fixed text
dump is the wrong material for that role. It can do both
jobs, but only badly. It cannot do both well at scale.
ss and the netlink sock_diag
family fixed the production case by adding a second
interface that one can actually ask. The old surface stays
for cat and grep. The new one
took over the actual workload. The cost of the transition
was a decade of slow netstat on production
servers, a long mailing-list conversation, and a
generation of administrators learning a new tool name. The
shape of the fix, twenty-two years on, looked exactly like
sysctl.
FreeBSD never had to choose, because FreeBSD never put a
non-askable file between its tools and the kernel in the
first place. The tool from 1983 still answers the question
on the workloads of 2026. The Linux ecosystem did the
right thing by adding ss. The fact that it
had to is the architectural lesson.
The Unix text interface is a tool one can question, not a
file one can read. netstat and
sockstat are the interface;
/proc/net/tcp only ever looked like one.
KISS, properly applied, knows the difference.
One tool from 1983 that still answers at scale. One tool that had to be written because the first one was too slow. Same question. Two architectures.