Every production incident is the same question asked under time pressure: what is this machine actually doing? The lineage that runs from /proc to eBPF — and, before it, from the counters ps and vmstat were reading in the 1970s — is half a century of answering that question with progressively less guessing. It is also the one corner of the Unix design story where the famous promise, everything is a file, did not merely survive contact with reality but paid for itself many times over. Then, having paid off, it was quietly abandoned. The modern observability stack is not a filesystem. It is a syscall, a bytecode verifier, a just-in-time compiler, and a handful of file descriptors pointing at maps.
The shape of the lineage is worth stating before the history: counters you read → syscalls you intercept → arbitrary safe code you inject. Each generation answered the previous one’s blind spot, and — this is the part engineers usually get backwards — each new generation was rarely more capable than the one before it. It was cheaper. ptrace-based tracing could already answer nearly any syscall question in 1991; it just cost too much to leave running. Every subsequent advance in this field has been an overhead argument wearing a capability costume.
And underneath both of those stories sits a third one, which is the reason this article exists rather than a timeline: a licence, not a benchmark, chose the tool the industry standardized on. DTrace was years ahead and lost Linux to a copyright-header incompatibility. That is not a footnote. It is the same causal shape that produced Unix itself.
Processes as Files: Killian, 1984, and a Debugging Ergonomics Problem
/proc originates in Eighth Edition Research Unix, implemented by Tom J. Killian, who presented “Processes as Files” at USENIX in June 1984 (Summer conference, Salt Lake City). Two precision notes before anything else, because both are routinely mangled: this is the Research Unix Edition line, not System V and not Linux — three unrelated version schemes that share no numbering — and the motivation was not observability at all.
The motivation was that ptrace(2) was miserable to use. It moved data between a debugger and its target one word at a time through a request-code interface, with a bespoke verb for every operation. Killian’s proposal was to stop inventing a private protocol for inspecting a process and instead make the process be something the system already knew how to talk to: a file. Then read(), write() and seek() work on it. Ordinary file permissions become the access-control model, with no new permission concept to design, document, or get wrong. A debugger becomes a program that opens a file.
Observability — the thing /proc is famous for — was a side effect of a debugging-ergonomics fix. Nobody set out to build a monitoring interface. They set out to delete an awkward syscall, and the generic interface they reached for turned out to be consumable by every tool that had ever been written and every tool that had not been written yet.
It is worth killing one myth here on the way past: this was not the Unix philosophy being applied. McIlroy’s canonical formulation appears in the 1978 Bell System Technical Journal Unix issue and is a post-hoc articulation of what had already worked. /proc postdates it by six years and came from a specific practical annoyance, not a creed.
The Abstraction Leaked Immediately: SVR4, ioctl(), and Plan 9’s Better Answer
The version most commercial Unixes actually inherited is not Killian’s. It is the rewrite: Roger Faulkner and Ron Gomes, “The Process File System and Process Model in UNIX System V,” USENIX, January 1991. It formalized the interface, gave each process a directory rather than a single file, and shipped into the System V world.
It was also driven through read(), write() and ioctl() — and that detail is the whole Unix abstraction story compressed into one interface. ioctl() is the escape hatch Unix reaches for whenever “everything is a file” turns out not to be true: a generic syscall that carries an opaque request code and an opaque argument, which is to say, a private protocol wearing a file descriptor as a disguise. The canonical case study for the file abstraction succeeding needed the canonical admission that it had failed, inside of seven years.
Plan 9 from Bell Labs — the original authors’ own attempt to finish what Unix left half-built — is the strongest evidence that they knew it. Its /proc is a directory per process (mem, status, ctl, text, regs) served by an actual file server speaking 9P, a protocol of roughly fourteen messages. Because the server is a server and not a kernel special case, you can mount another machine’s /proc and debug a process running on a different computer, on a different architecture, with a local debugger and no remote-debugging protocol at all. Per-process namespaces make that composable rather than a party trick. The Plan 9 papers describe the design directly, and it is a more coherent system than Unix by almost any measure. It went nowhere. That, too, is a recurring lesson in this material: the design that wins is the one that is cheapest to get, not the one that is best.
Linux Took the Fork in the Road — and /proc Became Everything
Linux added /proc in v0.97.3 (September 1992) and then did the thing that made it the interface the world actually reads: in v0.98.6 (December 1992) it extended /proc to non-process data.
That second step, three months later, is the fork in the road for the entire field. Solaris kept /proc scoped to processes and put system-wide statistics somewhere else entirely (kstat). Linux let /proc become a general kernel-state surface — /proc/meminfo, /proc/stat, /proc/loadavg, /proc/net/*, /proc/sys/* — and by doing so made the entire operating system’s internal state greppable with tools that were already two decades old when it shipped.
The consequence is hard to overstate and easy to take for granted. There was no SDK. No client library. No versioned RPC schema. No vendor agent to install, no daemon to run, no port to open, no authentication scheme to configure. Forty years of monitoring tooling — top, vmstat, sar, and, much later, Prometheus’ node_exporter — are, structurally, thin parsers sitting on top of text files. When a system architect asks the genuinely useful design question, can an operator answer a novel question about my service using tools I have never heard of?, /proc is the reference implementation of “yes.”
sysfs: One Value Per File, and the Retrofit of a Missing Discipline
The bill for that generosity came due as multi-field text blobs. /proc/stat grew positional columns; /proc/meminfo grew dozens of lines; individual files accumulated formats that only a dedicated parser could survive. /sys is the cleanup.
sysfs was written by Patrick Mochel during the 2.5 development series, and its name changed as its scope did: ddfs → driverfs (when the driver model merged) → sysfs (once kobjects generalized it past drivers). Mochel documented it in “The sysfs Filesystem” at the Ottawa Linux Symposium in 2005 (PDF, kernel.org). Its governing rule is a direct reaction to what /proc had become: one value per file.
That rule is worth stealing verbatim for any state-exposing interface you design. One value per file makes the schema self-describing, makes the parser trivial, makes the unit of change a single file rather than a shared format, and — critically — makes it impossible for a new field to break an existing consumer by shifting a column. /proc taught the industry that legibility is free. sysfs taught it that legibility without a shape discipline becomes an unmaintainable contract.
| Interface | Origin (dated) | Scope | Data shape | What a parser must know |
|---|---|---|---|---|
/proc, Eighth Edition | Killian, USENIX June 1984, Research Unix v8 | One file per process | The process itself — readable, writable, seekable, permission-checked | How to open(); a ptrace replacement, not a metrics feed |
/proc, SVR4 | Faulkner & Gomes, USENIX January 1991 | Directory per process; system stats live elsewhere (kstat on Solaris) | Structured files plus ioctl() control operations | The file layout and a private set of request codes |
/proc, Plan 9 | Bell Labs, served over 9P | Directory per process (mem, status, ctl, text, regs) | Text files served by a real file server — mountable across machines and architectures | A ~14-message file protocol, and nothing machine-specific |
/proc, Linux | v0.97.3 (Sept 1992); non-process data from v0.98.6 (Dec 1992) | Processes and global kernel state | Multi-field text, positional columns, per-file formats | A bespoke format per file — and which fields were added when |
sysfs | Mochel, 2.5 series; documented at OLS 2005 | Devices, drivers, buses, kobject-backed subsystems | One value per file | Nothing. cat is the parser |
A De-Facto ABI With a De-Jure Disclaimer
The kernel’s own /proc documentation states the tension outright. /proc “acts as an interface to internal data structures in the kernel,” and there is “no guarantee that every flag and associated mnemonic will be present in all further kernel releases.”
Read that as an engineer and it is an explicit refusal to promise stability. Read it as an operator and it is fiction, because nobody can reorder /proc/stat‘s columns without breaking a substantial share of the world’s monitoring in one release. The disclaimer is de jure; the ABI is de facto; the de facto one wins. This is exactly the trap of the internal JSON response that got scraped, the debug endpoint that became a dependency, the log line that someone’s alerting now regex-matches. A text interface that is trivially consumable will be consumed, and the ease that made it valuable is precisely what makes it permanent.
There is a second, sharper cost with modern consequences: /proc is not namespaced. A process inside a container reads /proc/meminfo and sees the host’s memory; it counts CPUs and sees the host’s CPUs. The actual limits live in cgroup v2, under /sys/fs/cgroup (memory.max, memory.current). This is why free and nproc lie inside a container, why language runtimes have had to grow explicit container-awareness rather than trusting the interface, and why capacity dashboards built on /proc inside containerized workloads can be confidently, silently wrong. The interface predates the isolation model by two decades and was never retrofitted to know about it.
The Overhead Ceiling: Why strace Never Became a Production Tool
Reading counters tells you how much. It never tells you who, why, or which code path. The next generation went after attribution by intercepting syscalls.
strace predates Linux: Paul Kranenburg wrote the first version for SunOS in 1991; Branko Lankester ported it to Linux; Rick Sladkey merged the branches in 1993 and pulled in features from SVR4’s truss (strace(1)). It answers questions counters cannot touch: which file did it fail to open, what did that syscall return, where did the process wedge.
And it is nearly unusable on a busy production system, because ptrace stops the traced process twice per syscall — once on entry, once on exit — with a context switch each way. The capability was never the constraint. The price was. Everything that follows in this story is an attempt to buy the same information for less.
| Generation | Representative tools | What it fixed | What it left unfixed |
|---|---|---|---|
| Counters (1970s–80s) | ps, vmstat (first appeared in 3BSD), iostat, sar (System V lineage) | Cheap, always-on aggregates you can sample forever | No attribution to a process, a file, or a code path |
| Syscall tracing (1990s) | truss (SVR4), strace, ltrace | Exactly which calls a process makes, and what they returned | ptrace stops the process twice per call; userland boundary only |
| Dynamic tracing (2004–) | DTrace, then SystemTap | Arbitrary kernel and user questions on a production box | Licence-bound (DTrace); out-of-tree and build-fragile (SystemTap) |
| PMCs and sampling (2009–) | perf / perf_events | Hardware truth: cycles, stalls, cache misses | Sampling is statistical — it cannot see rare events; PMCs often absent in VMs |
| Programmable in-kernel (2014–) | eBPF, bcc, bpftrace | Safe arbitrary code at almost any hook, aggregating in the kernel | Verifier complexity limits, kernel version skew, real authoring cost |
perf arrived on its own track — “Performance Counters for Linux” by Thomas Gleixner, Ingo Molnár and Peter Zijlstra, merged in 2.6.31 (2009) and renamed perf_event in 2.6.32 — and it is a different axis rather than a later rung. Sampling has bounded, tunable overhead and gives you a statistical picture; tracing sees every event and its cost scales with the event rate. Tracing every read() on a busy server is a way to take that server down.
DTrace and the Invention of Provable Safety in a Kernel
Bryan Cantrill, Michael Shapiro and Adam Leventhal presented “Dynamic Instrumentation of Production Systems” at USENIX ATC in 2004 (paper PDF); DTrace shipped as a product in Solaris 10 in 2005. Keep those two dates distinct — “DTrace was released in 2004” conflates a paper with a product, and the gap between the two is where a research prototype became something you could point at a customer’s box.
The syntax is not the contribution. The safety architecture is, and it remains the most interesting computer science in this entire lineage. Cantrill’s non-negotiable requirement was blunt: “DTrace must not be able to accidentally induce system failure.” Not “should rarely.” Not “with care.” The design follows from taking that literally:
- The D language has no backward branches. User-defined loops are therefore not restricted, or bounded, or checked — they are impossible to express. Termination is guaranteed by the grammar rather than proved by an analysis.
- Instrumentation runs in a virtual machine (DIF), not natively. Runtime faults — divide by zero, a misaligned load, an illegal dereference — are caught by the VM, which refuses the operation and aborts that enabling instead of corrupting the kernel. Cantrill argued the point explicitly: once you let user-authored code execute natively in kernel context, you can no longer distinguish a legal store from kernel corruption, or detect an infinite loop at all.
- Probes are provider-chosen sites, not arbitrary instruction addresses that a user picks and hopes about.
- The paper’s overhead claim is narrow and precise: zero probe effect when probes are disabled. That is the claim worth repeating, and it is not the same claim as “tracing is free.”
The transferable pattern is one sentence: restrict expressiveness until the property you need becomes decidable, then recover performance somewhere else. You will recognize it immediately in WASM sandboxes, in edge-worker isolates, in SQL query planners, and in every plugin API that refused to accept a general-purpose language.
A Licence, Not a Benchmark, Chose the Winner
DTrace was open-sourced with OpenSolaris under the CDDL. The CDDL is not GPL-compatible. Therefore DTrace could not be merged into the Linux kernel — not because of a technical objection, not because of a performance regression, not because anyone benchmarked it and found it wanting, but because two copyright headers could not legally be combined into one distributable binary.
Follow the consequences, because they are enormous and entirely non-technical:
- FreeBSD had no such conflict and simply shipped DTrace, in 7.1 (2009). macOS shipped it too. The tool worked fine. The obstruction was legal, not engineering.
- Linux built its own instead.
kprobeslanded in-tree in 2004. SystemTap followed from Red Hat — a technology preview in RHEL 4 Update 2 in 2005, 1.0 in 2009, with IBM, Intel, Hitachi and Oracle contributing. LWN was documenting the resulting mood as “DTrace envy” by 2007. - eBPF then arrived from an unrelated direction and won, for reasons covered in the next section.
- Oracle relicensed DTrace under GPLv2 in February 2018 — reported at the time by The Register, among others. By then the argument was over. eBPF was in the mainline kernel, had a verifier, a JIT, and a growing tool ecosystem. The legal obstacle was removed roughly a decade after it stopped mattering.
State that carefully, because it is a licensing fact and not a moral one: nothing here requires a view about Sun’s or Oracle’s motives, and the CDDL was not written to spite Linux. The observable outcome is what matters. A choice made in a licence header determined which observability architecture a generation of engineers would learn, deploy, and build careers around.
This is a pattern in Unix history rather than an incident. The 1956 antitrust consent decree barred AT&T from any new business outside common-carrier communications; the practical effect was that Unix could not be sold as a supported product line. It was still licensed commercially — roughly US$20,000 for a Sixth Edition licence, and never free — but AT&T’s own stated policy was “no advertising, no support, no bug fixes, payment in advance.” So it went to universities cheaply, with source, and a generation learned operating systems by reading it. When the 1982 Modification of Final Judgment finally freed AT&T to commercialize, prices rose, source access closed, and three separate replacement efforts followed — GNU, BSD, and Linux — every one of which inherited the interface. A legal document with no technical content in it determined the shape of the world’s software. The DTrace story is that same mechanism at a smaller scale, decades later, and to anyone who thinks “we can always relicense it later,” the timeline offers a precise answer: legally true, practically false, and the gap between those two is measured in years you do not get back.
eBPF: A 1993 Packet Filter That Ate the Kernel
eBPF’s ancestor was not built for tracing, and neither was eBPF. Steven McCanne and Van Jacobson published “The BSD Packet Filter” at USENIX Winter 1993 (conference record) — a tiny in-kernel register machine whose entire job was letting tcpdump reject uninteresting packets in the kernel instead of copying every one of them to userspace.
In 2014, Alexei Starovoitov (with Daniel Borkmann) extended that machine into what is now called eBPF: 64-bit registers, maps for sharing state with userspace, a JIT compiler, and the bpf() syscall. It was merged in Linux 3.18. Tracing was downstream of both generations — a general-purpose safe-execution engine that got pointed at instrumentation because it happened to be the right shape for it.
The safety bet is the same one DTrace made, executed with different mechanics. The verifier performs a DAG check that rejects cycles outright, then symbolically executes every reachable path, tracking register types and value ranges, pruning at fixpoints, and proving termination before the program is allowed to load. Bounded loops are accepted only when an exit condition provably fires, and every path permutation counts against a complexity budget — which is why real programs hit the instruction limit and reach for bpf_loop() (up to roughly 8M iterations, verified once). Programs that survive are then JIT-compiled to native machine code. The safety tax is paid once, at load time, rather than on every event — which is precisely the trade strace could never make.
Two honest qualifications, because this area attracts marketing. First, the verifier does not make eBPF “safe” in the security sense. It proves termination, bounded memory access, and the absence of certain classes of leak. It has had CVEs. Write “verified,” state the scope, and never write “cannot harm the system.” Second, “eBPF has near-zero overhead” is a slogan, not a measurement. Tracing cost scales with event rate no matter how good the runtime is; the only overhead claim in this lineage with a documented, narrow scope is DTrace’s zero probe effect when probes are disabled.
| DTrace | SystemTap | eBPF | |
|---|---|---|---|
| First shipped | Paper USENIX ATC 2004; product in Solaris 10, 2005 | Tech preview RHEL 4 U2, 2005; 1.0 in 2009 | Merged in Linux 3.18, 2014 |
| Safety mechanism | No backward branches in D, so loops are inexpressible; DIF virtual machine catches runtime faults and aborts the enabling | Translates a script to C and builds a loadable kernel module — safety rests on the toolchain and matching kernel headers/debuginfo | In-kernel verifier: cycle rejection, per-path symbolic execution, provably-bounded loops, complexity budget |
| Where the cost is paid | Probe design time (provider-chosen sites) plus per-operation VM checks | Build time — needs a matching kernel build environment | Program load time, once; then JIT to native code |
| Overhead claim that is actually documented | Zero probe effect when probes are disabled | — | — (event-rate-dependent; treat vendor figures as unsourced until they are) |
| Licence position | CDDL — GPL-incompatible until Oracle’s February 2018 GPLv2 relicensing | GPL, but historically out of tree | In the mainline kernel tree |
| Where it runs | Solaris/illumos, FreeBSD (7.1, 2009), macOS | Linux, with debuginfo available | Linux, mainline |
The frontends, notably, came from individuals rather than vendors: bcc was created by Brenden Blanco (then at PLUMgrid, under iovisor), and bpftrace began as Alastair Robertson’s spare-time project before Brendan Gregg and others grew it into what Gregg openly calls “DTrace 2.0 for Linux.” A decade and more after the CDDL closed the direct route, Linux had the thing it had been denied — built, of all things, out of a packet filter.
The Numbers That Outlived Their Definitions
There is a second failure mode here that has nothing to do with mechanisms and everything to do with reading. /proc made kernel state legible; it did not make it comprehensible. Several of the most-read numbers on any Unix system mean something other than what their names suggest, and in every case the divergence is documented, dated, and traceable to a specific change.
The clearest example is the one everybody looks at first. Load average is not a CPU metric on Linux, and Unix did not invent it — it comes from TENEX, where RFC 546 (August 1973) documents it as a count of runnable processes, which is pure CPU demand. Linux deliberately broadened the definition: Brendan Gregg traced the change to a patch by Matthias Urlichs dated 29 October 1993, applied in Linux 0.99 patchlevel 14, arguing that processes waiting on non-interruptible I/O “also consume resources.” Linux has counted TASK_UNINTERRUPTIBLE tasks alongside runnable ones ever since. That is a definition divergence, argued and intentional, not a bug — but it means a load average of 16 on an 8-core box may indicate CPU saturation, a stuck NFS mount, a slow disk, or a kernel lock, and the number cannot tell you which. The replacement, notably, names its resource in the path: Pressure Stall Information by Johannes Weiner, landed in Linux 4.20, exposes /proc/pressure/{cpu,memory,io} with some and full stall percentages.
The same shape repeats across the file. MemFree trends toward zero on any healthy system because page cache is reclaimable by design, and the folk correction of adding free + cached is wrong in the kernel’s own words — Rik van Riel’s commit 34e431b0ae39, “/proc/meminfo: provide estimated available memory”, added MemAvailable in Linux 3.14 (2014) because free-plus-cached “was fine ten years ago, but is pretty much guaranteed to be wrong today.” Read available, and keep the kernel’s own noun for it: an estimate. %CPU measures occupancy, not work, since a thread stalled on main memory is still on-CPU — Gregg’s worked example puts an IPC of 0.78 on a four-wide issue core at roughly 19.5% of peak instruction throughput. And %iowait is an idle state, not busy time: it counts a CPU idle while at least one I/O is outstanding, so a high value may only mean nothing else was runnable, and a low one never proves storage is healthy.
The pattern is identical in every case, and it is why this belongs in an article about interfaces: a widely adopted metric accumulates users far faster than it accumulates readers of its definition. After that you cannot fix it — you can only ship something beside it, which is exactly what PSI and MemAvailable are. The design rule that falls out is prefer metrics whose name states the resource and the unit; the same discipline that makes a network measurement useful is what load average and %CPU never had.
The oldest tool in the box states the boundary most cleanly. POSIX defines time‘s -p output as real, user and sys, where real is elapsed wall-clock time and the other two come from times(), including reaped children. So real ≫ user + sys is the operating system telling you, in three numbers and a 1970s syscall, that your process was not running — while telling you nothing at all about why. Blocked on I/O, descheduled under contention, waiting on a lock, and stalled on a page fault all produce the identical signature. That gap is precisely the off-CPU time the entire lineage from ptrace to eBPF exists to explain.
What an Architect Should Steal From Fifty Years of This
1. Expose internal state through the most generic interface your platform already has. Not “use a filesystem” — the transferable rule is that the value of /proc came from being consumable by tools its authors had never heard of and could not anticipate. No SDK, no client library, no versioned schema, no agent. If answering a novel question about your system requires a bespoke client, you have built a ptrace, not a /proc.
2. Any easily-consumed interface becomes a permanent contract, whether you promised one or not. The kernel documents no stability guarantee for /proc and is bound by it anyway. Assume a disclaimer buys you nothing once consumers exist, and adopt sysfs’s one value per file discipline early, when the schema is still yours to shape.
3. Ask what it costs to leave observability on, not what it can see. Every generation in this lineage answered an overhead constraint, not a capability one. The right question of any observability tool or vendor is never “what can it see” but “what does it cost to leave it seeing that, at my event rate.”
4. Restrict expressiveness until the property you need is decidable, then compile. DTrace banned backward branches; eBPF proves termination in a verifier and JITs the result. Same bet, different mechanics, and it generalizes to every sandbox, plugin API, and embedded query language you will ever design.
5. Treat licences as architecture with a decade-scale blast radius. CDDL versus GPLv2 chose the observability stack of an entire industry, and the 2018 relicensing proved the point by arriving far too late to change anything. “We can relicense later” is legally true and practically false.
6. Grant the win where it was real. This series is hard on “everything is a file” — sockets cannot be open()ed by path, ioctl() is the standing admission of failure, netlink is another. /proc is the exception that proves how rarely the abstraction held, and it deserves the credit fully. It is also worth noticing where the story ended up: eBPF is a syscall, with file descriptors and maps, and it is not a filesystem. The industry took the lesson of /proc — make state legible — and left behind the mechanism that taught it.
❓ Frequently Asked Questions
Is /proc part of POSIX or the Single UNIX Specification?
No. The standards specify the times() interface and the time utility, not a process filesystem — which is exactly why every Unix that has one has a different one. Killian’s Eighth Edition version, SVR4’s ioctl()-driven rewrite, Plan 9’s 9P-served directories, and Linux’s general kernel-state surface are four incompatible designs sharing a mount point name. Portable code cannot assume any of them.
Why did DTrace never make it into Linux?
Licensing, not merit. DTrace was released under the CDDL, which is not GPL-compatible, so the code could not be merged into the kernel and distributed. Linux built kprobes (in-tree 2004) and then SystemTap instead, and eventually eBPF. FreeBSD, which had no licence conflict, simply shipped DTrace in 7.1 in 2009 — the clearest available evidence that the obstacle was legal rather than technical.
Did Oracle’s 2018 GPLv2 relicensing of DTrace change anything?
Effectively no. By February 2018, eBPF had been in the mainline kernel since 3.18 in 2014, with a verifier, a JIT, and the bcc/bpftrace tooling ecosystem growing around it. Removing a legal obstacle roughly a decade after it stopped being the deciding factor does not reverse an adoption curve — which is the practical case against assuming any licence can be fixed later.
Is eBPF actually safe to run in production?
“Verified” is the accurate word, and the scope matters. The verifier proves termination, rejects cycles, bounds loops, and checks memory access before a program loads; it does not make eBPF invulnerable, and it has had CVEs. Treat it as a strong, narrow guarantee — the program will not loop forever or read arbitrary kernel memory — rather than a claim that a loaded program cannot harm the system.
Why does free inside a container show the host’s memory?
Because /proc is not namespaced. /proc/meminfo and the CPU count both report host-wide values regardless of which container asks, while the actual limits live in cgroup v2 under /sys/fs/cgroup (memory.max, memory.current). Any tool that derives capacity from /proc inside a container is reading the wrong file — the interface predates the isolation model by two decades.
What should I use instead of load average?
Pressure Stall Information, where available: /proc/pressure/{cpu,memory,io} reports some and full stall percentages per resource, so the number names the thing that is saturated. Load average is not broken — Linux deliberately counts uninterruptible-sleep tasks as well as runnable ones, following Urlichs’ October 1993 patch — but one dimensionless number covering CPU, disk and lock waits cannot tell you which of the three you are looking at, and PSI can.
📚 Further Reading
- What Is a Container? The Idea That Powers Modern Software (and AI) — why a namespaced process still reads an un-namespaced
/proc, and where the real limits live. - Observing Distributed Workflows: How to Tell a Saga Is Healthy — the same legibility problem one layer up, where the “process” is spread across services.
- Measuring Network Performance — the companion discipline of naming the resource, the unit, and the window before trusting a number.
- DORA Metrics for AI Coding Agents: Stop the Gaming — what happens to any widely adopted metric once it accumulates users faster than readers of its definition.
- The Best Books on Distributed Systems (Beginner to Expert) — for the broader reading on measuring systems that are too big to watch directly.
📚 Recommended Resources
Disclosure: some links below are Amazon affiliate links. If you buy through them, I earn a small commission — at no extra cost to you — which helps keep this site running.
Before any of these: two of the best references on this material are free and canonical. Operating Systems: Three Easy Pieces is a complete, freely readable OS textbook covering the process, memory and I/O models every metric in this article ultimately reports on. And the POSIX.1-2024 Base Specifications (Issue 8) are free to read in full — that is where you check what times(), time, and the rest of the interface actually promise, rather than what a man page on one system happens to say.
Systems Performance (2nd ed.)
Brendan Gregg · Book
The single most useful book for the second half of this article. It is the standard reference for the USE method, off-CPU analysis, and the load-average and %CPU corrections covered above — Gregg is the person who traced the October 1993 patch that redefined Linux’s load average, and this is the long-form version of that work applied across every resource in the system.
BPF Performance Tools
Brendan Gregg · Book
The applied companion: dozens of bcc and bpftrace tools, what each one hooks, and what its overhead actually depends on. Read it after the verifier section here if you want to go from understanding why eBPF is safe to load to knowing what to load — and, more usefully, what it costs to leave running.
Advanced Programming in the UNIX Environment (3rd ed.)
W. Richard Stevens & Stephen A. Rago · Book
The systems-programming reference underneath everything /proc exposes: process lifecycle, file descriptors, signals, and the times() accounting that time(1)‘s three numbers come from. If the real ≫ user + sys gap in this article is the part you want to chase, this is where the primitives are documented properly. (Note the edition — the 2nd edition is a separate, older listing.)
The Design and Implementation of the FreeBSD Operating System (2nd ed.)
Marshall Kirk McKusick, George V. Neville-Neil & Robert N. M. Watson · Book
The kernel-internals counterweight to a Linux-centric view — and directly relevant here, because FreeBSD is the system that had no licence conflict and simply shipped DTrace in 7.1 in 2009. Reading how a second, independently maintained Unix kernel structures the same subsystems is the fastest way to tell which parts of Linux’s design are necessary and which are historical. This is the current edition; no third exists.