Somewhere in every organization that runs servers there is an alert wired to the wrong number, and it is almost always this one. Linux free vs available memory is not a stylistic preference between two columns of the same tool — they answer two genuinely different questions, and only one of them is the question an operator is actually asking. MemFree reports how much RAM the page allocator is currently holding in reserve. MemAvailable estimates how much a new process could obtain before the system starts swapping. On a healthy long-running box those two numbers routinely differ by an order of magnitude, and the folk workaround people reach for instead — add free and cached together — has been documented as wrong by the kernel itself since 2014.
This is not a trivia correction. It is a case study in a pattern that shows up in every long-lived interface: the number that is cheap to compute and the number that answers your question are rarely the same number. Interfaces ship the cheap one first, because it falls out of a data structure that already exists, and then spend decades adding the useful one alongside it — without ever being able to remove the misleading original, because by then everything parses it.
Free Memory Is Wasted Memory
Start with the fact that makes the whole subject counterintuitive: on a well-behaved Linux server, MemFree trending toward zero is the system working correctly.
RAM that holds nothing is RAM you paid for and are not using. The kernel therefore fills it with the page cache — file-backed pages read from or written to disk, kept resident so the next access is a memcpy instead of an I/O. That cache is reclaimable: when a process needs anonymous memory and the allocator is short, clean file-backed pages are dropped without ceremony and the memory is handed over. The cost of having cached the wrong thing is one page fault. The cost of not caching is every future read of that file.
An operating system that reports large amounts of free memory after hours of uptime is not being frugal. It is failing to use hardware you already bought. This is why the “Linux ate my RAM” support ticket is perennial and why the correct answer is always the same: nothing was eaten, the cache is doing work, and the number you are reading does not mean what you think it means.
The honest qualification, which most explanations skip: free memory is not entirely waste. The allocator needs headroom. Zone watermarks exist so that allocations which cannot block — interrupt context, GFP_ATOMIC — have something to draw on without waiting for reclaim, and so that reclaim can start before the system is already stalled. Higher-order allocations need physically contiguous runs, which fragmentation can make unavailable even when the byte count looks fine. So a small floor of genuinely free memory is load-bearing. What is not load-bearing is the several gigabytes of it that a freshly booted machine reports and that a real workload will, correctly, consume within the hour.
Linux free vs available Memory: The Fields That Actually Matter
Everything user-space reports comes out of /proc/meminfo, which the kernel documents field by field in the /proc filesystem documentation. The definitions matter more than the tooling, because every tool is a thin parser over these lines.
| Field | Kernel’s own definition | What it means operationally | Reclaimable? |
|---|---|---|---|
MemTotal | “Total usable RAM (i.e. physical RAM minus a few reserved bits and the kernel binary code)” | Your denominator. Never equals the DIMM total. | — |
MemFree | “Total free RAM” | Allocator headroom right now. Trends to a small floor on any busy box. | Already free |
MemAvailable | “An estimate of how much memory is available for starting new applications, without swapping.” | The number you actually want. An estimate — the kernel says so. | — |
Buffers | “Relatively temporary storage for raw disk blocks shouldn’t get tremendously large (20MB or so)” | Block-device caching. The kernel’s own note tells you not to expect it to be large. | Mostly yes |
Cached | “In-memory cache for files read from the disk (the pagecache) as well as tmpfs & shmem” | The trap. Includes tmpfs and shared memory, which are not droppable. | Partly |
Shmem | “Total memory used by shared memory (shmem) and tmpfs” | Counted inside Cached. Backed only by swap, so it can never simply be dropped. | No (swap only) |
SReclaimable | “Part of Slab, that might be reclaimed, such as caches” | Dentry and inode caches. Real, reclaimable memory that Cached omits entirely. | Yes |
SUnreclaim | “Part of Slab, that cannot be reclaimed on memory pressure” | Kernel data structures. Gone until the objects are freed. | No |
Dirty | “Memory which is waiting to get written back to the disk” | Reclaimable only after writeback completes — a latency, not a byte count. | Eventually |
Mlocked | “Memory locked with mlock()“ | Pinned by userspace. Databases and crypto daemons do this deliberately. | No |
SwapCached | “Memory that once was swapped out, is swapped back in but still also is in the swapfile” | Pages with a free ticket back out. Cheap to reclaim. | Yes |
Read that table once and the naive arithmetic collapses on its own. free + buffers + cached is wrong in both directions simultaneously: it counts memory that cannot be reclaimed (tmpfs, shared segments, mlocked regions, dirty pages that still need a disk round-trip) and it omits memory that can be (reclaimable slab, which is nowhere in Cached).
The 2014 Commit That Named the Problem
The best short explanation of any of this was not written by a blogger. It is a kernel commit message. In Linux 3.14 (2014), Rik van Riel added MemAvailable to /proc/meminfo in commit 34e431b0ae39, and stated the case in one sentence: applications trying to size themselves against available memory “generally do this by adding up ‘free’ and ‘cached’, which was fine ten years ago, but is pretty much guaranteed to be wrong today.”
The patch’s inline comments spell out what the estimate does instead, and they are worth reading as the actual specification:
- “Free memory cannot be taken below the low watermark, before the system starts swapping.” So the estimate begins at
MemFreeminus the low watermark, not atMemFree. - “Not all the page cache can be freed, otherwise the system will start swapping. Assume at least half of the page cache, or the low watermark worth of cache, needs to stay.” The cache contribution is deliberately discounted rather than counted whole.
- Reclaimable slab is then added — and capped, because part of it is in use at any moment and cannot actually be freed.
MemFree | free + buffers + cached | MemAvailable | |
|---|---|---|---|
| Counts idle pages | ✅ | ✅ | ✅ (minus the low watermark) |
| Counts page cache | ❌ | ✅ all of it | Discounted, not counted whole |
| Excludes tmpfs / shmem | n/a | ❌ counts it as free | Drawn from the file LRU lists, not from Cached |
| Counts reclaimable slab | ❌ | ❌ misses it entirely | ✅ capped |
| Respects the swap threshold | ❌ | ❌ | ✅ |
| Cost to compute | One counter read | Three counter reads | A short walk over zones and LRU lists |
| Failure mode | Alerts fire on a healthy box | Silently optimistic | Optimistic under a large tmpfs |
That last row is the part to internalize, and the kernel is explicit about it: MemAvailable is “an estimate.” It is not a promise. Shared-memory pages sit on the file LRU lists the estimate draws from, so a machine with a very large tmpfs can still read optimistically. Heavy dirty-page load makes the reclaimable portion available only after writeback, which is a latency the byte count does not express. Treat MemAvailable as a well-constructed heuristic that is right far more often than the alternatives, not as a guarantee that an allocation of that size will succeed.
Reading the Numbers Without Being Lied To
The discipline worth importing here is Brendan Gregg’s USE method — for every resource, check utilization, saturation, and errors — because it enumerates the resources first and then goes looking for tools, rather than enumerating whatever your agent happens to export and calling the result a dashboard. Applied to memory, that reorders the tooling below into something like a priority list, with the saturation signal at the top rather than the level.
free -h. The modern tool already does the right thing, and its own man page documents the change. Per free(1), the used column is now “calculated as total - available” — not total - free. The available column is MemAvailable from /proc/meminfo, described there as taking “into account page cache and also that not all reclaimable memory slabs will be reclaimed due to items being in use,” present on kernels 3.14 and up and emulated on 2.6.27+. shared is Shmem. buff/cache is the sum of buffers and cache. If your dashboard predates that, it is reporting a column the tool itself stopped believing in.
vmstat 1. The columns that matter for memory are not in the memory group. They are si and so — pages swapped in and out per second. Sustained non-zero so is the machine telling you it has run out of reclaimable memory and is paying for it in latency. free being small tells you nothing; so being non-zero tells you everything. The b column (processes in uninterruptible sleep) is the companion signal.
smem and PSS. RSS double-counts: every process mapping a shared library reports the full size of that library in its own RSS, so summing RSS across processes routinely exceeds physical memory. /proc/PID/smaps exposes PSS — proportional set size, where shared pages are divided among the processes mapping them — and smem is the standard front-end. If you are attributing memory to services rather than measuring the machine, PSS is the number that adds up.
PSI — the modern saturation signal. Levels tell you where you are; pressure tells you what it is costing. Pressure Stall Information, authored by Johannes Weiner and merged in Linux 4.20, exposes /proc/pressure/memory with two lines. some is “the share of time in which at least some tasks are stalled on a given resource”; full is “the share of time in which all non-idle tasks are stalled on a given resource simultaneously,” each averaged over 10, 60, and 300 seconds. A full figure climbing above zero means the machine is spending real wall-clock time doing reclaim instead of work — which is the thing you actually wanted to know when you went looking at free. LWN’s coverage as the patches landed frames the motivation the same way: existing metrics tell you a resource is busy, not that work is being lost waiting for it. The same files exist per-cgroup as memory.pressure, which is what makes PSI the one memory signal that means the same thing inside a container as outside one.
This is the same move the kernel made with load average, which is worth naming because it is the identical failure resolved the identical way. Linux load average stopped being a CPU metric with a patch by Matthias Urlichs dated 29 October 1993, applied in Linux 0.99 patchlevel 14, which added TASK_UNINTERRUPTIBLE tasks to the count on the argument that processes blocked on fast I/O also consume resources. That was a deliberate definition change, not a bug — and the metric itself was inherited, not invented here: it comes from TENEX, documented in RFC 546 (1973). The number could not be fixed, only supplemented. PSI is the supplement, for load average and for memory alike.
Containers: Where free Stops Being Misleading and Becomes Simply Wrong
Here is the recurring production incident, and it has caught every organization that runs containers at least once.
/proc/meminfo is not namespaced. A process inside a container reads the host’s numbers. A service in a 4 GiB container on a 256 GiB host runs free and sees roughly 250 GiB available. Anything that sizes itself from that number — a connection pool, a JVM heap, a worker count, an in-process cache, a GOMEMLIMIT computed at startup — sizes itself for a machine it does not have. The container is then OOM-killed at 4 GiB while every host-level dashboard shows a machine with plenty of headroom, which is exactly the observation that sends people down the wrong debugging path for a day.
The real limit lives in cgroup v2, under /sys/fs/cgroup, and the kernel’s cgroup v2 documentation defines the files precisely.
MemFree (/proc/meminfo) | MemAvailable (/proc/meminfo) | memory.current (cgroup v2) | |
|---|---|---|---|
| Scope inside a container | The host | The host | This cgroup and its descendants |
| What it measures | Idle host RAM | Estimated host allocatable RAM | “The total amount of memory currently being used by the cgroup and its descendants” |
| Includes page cache? | No | Discounted | Yes — the cgroup’s file cache counts against it |
| Tells you the limit | No | No | No — read memory.max |
| Enforcement | None | None | memory.max = hard limit → OOM; memory.high = throttle + heavy reclaim |
| Safe to size a service from? | No | Not inside a container | Only with memory.stat alongside it |
The critical detail is in that “includes page cache” row, and it is the reason this article circles back on itself. memory.current counts the cgroup’s own page cache. A container that has read a few gigabytes of files will show memory.current close to memory.max while being nowhere near trouble, because most of that is reclaimable cache which will be dropped the instant anonymous memory needs the space. Alerting on memory.current / memory.max is therefore the container-era restatement of alerting on MemFree — the same misreading, one abstraction layer up.
And the punchline for anyone hoping the kernel already solved this: there is no memory.available. The cgroup v2 memory controller documents memory.current, memory.min, memory.low, memory.high, memory.max, memory.reclaim, memory.peak, memory.oom.group, memory.events, memory.stat, memory.numa_stat, the swap and zswap files, and memory.pressure. The useful global number arrived in 2014 and the cgroup analogue still has not. Computing headroom yourself means subtracting the reclaimable part out of memory.stat — file, which the docs define as memory “used to cache filesystem data, including tmpfs and shared memory,” inactive_file, and slab_reclaimable, “part of ‘slab’ that might be reclaimed, such as dentries and inodes” — which is to say you get to re-derive van Riel’s 2014 estimate by hand, per cgroup, and get the tmpfs caveat wrong in exactly the same place.
The practical rules that fall out:
- Read the cgroup files, not
/proc/meminfo, for anything inside a container.memory.max,memory.current,memory.stat,memory.events,memory.pressure. - Use
memory.pressure, not a level, for alerting. PSI is per-cgroup and does not care whose page cache it is. - Prefer a runtime that already knows. OpenJDK’s
UseContainerSupport(JDK-8146115) has been on by default since JDK 10 and was backported to 8u191; Go’sGOMEMLIMIT, added in Go 1.19 (2022), gives the runtime a soft memory ceiling you can derive from the cgroup at startup. Anything sizing itself fromsysconforfreeinside a container is sizing itself from a lie. lxcfsexists specifically to synthesize a cgroup-aware/proc/meminfoinside a container, which is a workaround for the namespacing gap rather than a fix to it.- Know which limit you set.
memory.highthrottles the cgroup and applies heavy reclaim pressure;memory.maxis where the OOM killer gets invoked. A service tuned againsthighdegrades; a service that hitsmaxdies.
“The Box Had Free Memory, but the OOM Killer Fired”
This complaint is usually true and almost never means what the person saying it thinks.
The global OOM killer runs only after reclaim has failed to satisfy an allocation. Its selection logic is deliberately blunt, and the comment above oom_badness() in mm/oom_kill.c says so outright: “The heuristic for determining which task to kill is made to be as simple and predictable as possible. The goal is to return the highest value for the task consuming the most memory to avoid subsequent oom failures.”
The score is a sum of three things: the task’s resident set size, its swap entries, and its page-table bytes divided by page size. oom_score_adj is then scaled by total RAM and added, so the tunable is a proportional nudge rather than an absolute override — with one exception, OOM_SCORE_ADJ_MIN, which removes a task from consideration entirely. Then select_bad_process() does exactly what its comment says: “Simple selection loop. We choose the process with the highest number of ‘points’.”
Note what is not in that calculation: which process requested the allocation that failed, which process grew fastest, or which process is least important to your business. The kernel kills the biggest, because killing the biggest is most likely to prevent a second OOM in ten seconds. Your database is the biggest. That is the entire mystery behind “why did it kill Postgres instead of the leaking script.”
So when the box “had free memory,” check these four things in order:
- A cgroup limit, not global RAM. By far the most common answer.
memory.maxwas reached in one cgroup while the host had hundreds of gigabytes idle. This kill may not look like a global OOM at all — checkmemory.eventsfor theoomandoom_killcounters in the offending cgroup, and note thatmemory.oom.groupturns a single kill into the whole cgroup going down together. - A NUMA node, not the machine. An allocation bound to one node fails when that node is exhausted, regardless of what the other socket has free.
- The allocation context. A
GFP_ATOMICallocation cannot sleep, so it cannot wait for reclaim; it either finds memory in the reserves the watermarks protect or fails on the spot. Higher-order allocations need physically contiguous pages, which fragmentation can deny while the total byte count looks comfortable. - The watermark, not zero. Reclaim begins at the low watermark and allocations start failing well above
MemFree == 0. “It still showed 800 MB free” and “it was out of memory” are entirely compatible statements.
The Transferable Lesson: Cheap Numbers Ship First, and Never Leave
Strip out the kernel specifics and a general shape remains, one worth carrying into any interface you design.
MemFree shipped first because it was nearly free to produce. The allocator already tracked its own free lists; exposing the count was a read of a variable that existed anyway. MemAvailable could not ship first, because computing it requires answering a policy question — how much of this cache do we consider genuinely droppable, and how much headroom does reclaim need to avoid thrashing? — and policy questions do not fall out of data structures. So the cheap number arrived roughly two decades before the useful one, and the useful one had to be added beside it rather than in place of it, because by 2014 every monitoring agent, init script, and provisioning tool on earth was already parsing the cheap one.
That asymmetry is the lesson. You can add a field. You cannot remove one. The kernel’s own /proc documentation carries a no-stability guarantee while the practical reality is that nobody can reorder a column without breaking half the world’s monitoring — a de facto ABI with a de jure disclaimer. Every metric you expose in version one is permanent, including the ones that turn out to answer the wrong question.
The same pattern has run three times in this one corner of Linux. Load average, inherited from TENEX in 1973, was cheap to sample and had to be supplemented by PSI in 2018 rather than corrected. %CPU measures occupancy — a counter that already existed — and needed instructions-per-cycle beside it to distinguish a busy core from a core stalled on memory. MemFree measured the allocator’s bookkeeping and needed MemAvailable beside it to answer the operator’s question. In all three cases the original number is still there, still the default, still wired into somebody’s alert.
The design rule that falls out is narrow and useful: prefer metrics whose name states the resource, the unit, and the question being answered. MemFree names a state. MemAvailable names a question. some/full pressure names a resource and a cost in time. When you are tempted to expose the number your implementation makes easy, spend the extra afternoon working out what the consumer will actually divide it by — because whichever one you ship, you are shipping it forever.
❓ Frequently Asked Questions
Can MemAvailable be wrong?
Yes, and the kernel says so in its own documentation by calling it “an estimate of how much memory is available for starting new applications, without swapping.” It is derived from reclaimable file-LRU pages and reclaimable slab with deliberate safety discounts. It reads optimistically on systems with a very large tmpfs, because shared-memory pages sit on the file LRU lists the estimate draws from, and it does not express the writeback latency that dirty pages impose before their memory is actually usable. It is a well-constructed heuristic that beats every alternative — not a guarantee that an allocation of that size will succeed.
Should I run echo 3 > /proc/sys/vm/drop_caches to free up memory?
Only for benchmarking, where you deliberately want a cold cache. On a production system it is counterproductive: you are discarding cached data the kernel would have released automatically the moment anything needed the space, and the machine then pays to read all of it back from disk. Dropping caches makes MemFree look better and makes the server slower. If a dashboard is the reason you want to do it, fix the dashboard.
Why does free show almost no free memory on a perfectly healthy server?
Because that is what a healthy long-running server looks like. The kernel fills otherwise-idle RAM with page cache, which is reclaimable on demand, so MemFree settles to a small floor above the zone watermarks. Read the available column instead — it is MemAvailable, and modern free even computes its used column as total - available rather than total - free for exactly this reason.
How do I get correct memory numbers inside a container?
Stop reading /proc/meminfo, which is not namespaced and reports the host. Read the cgroup v2 files under /sys/fs/cgroup: memory.max for the limit, memory.current for usage, memory.stat to separate reclaimable file cache and slab from anonymous memory, memory.events for OOM counters, and memory.pressure for PSI. There is no memory.available, so headroom has to be derived by subtracting the reclaimable portion yourself. Prefer runtimes that already do this — OpenJDK’s UseContainerSupport, Go’s GOMEMLIMIT — over anything that sizes itself from free.
Why did the OOM killer take out my database instead of the process that was leaking?
Because it does not look for a culprit. oom_badness() scores each task by resident set size plus swap entries plus page-table bytes, adjusted by oom_score_adj, and select_bad_process() picks the highest score. The stated goal in the kernel source is to kill the task consuming the most memory so a second OOM does not follow immediately. Your database is the largest resident process on the box, so it wins that contest every time. Use oom_score_adj to protect it — and if the kill happened inside a container, the real cause is memory.max, not the host running out of RAM.
Is sustained swap usage a problem, or is swap being used at all the problem?
Swap being occupied is not a problem — pages that were idle at some point in the past and got written out cost nothing while they sit there. Swap being actively trafficked is the problem. Watch si/so in vmstat 1, and watch /proc/pressure/memory. A large SwapCached value with no ongoing swap-in or swap-out means the kernel made a good decision earlier and is holding the receipts.
📚 Further Reading
- What Is a Container? The Idea That Powers Modern Software (and AI) — the namespace and cgroup mechanics underneath the container-blindness problem in this piece, from the ground up.
- Postgres as a Vector Database: What Benchmarks Say — a workload where the page-cache-versus-
shared_buffersdistinction decides real benchmark numbers, not just dashboard readings. - Observing Distributed Workflows: How to Tell a Saga Is Healthy — the same “which signal actually answers the question” discipline applied to workflow health rather than machine resources.
- How to Run AI Models Locally With Docker (a Private ChatGPT on Your Own Machine) — where container memory limits stop being theoretical, since model weights are the thing that hits
memory.maxfirst. - Why Is My Computer So Slow? Diagnosing and Fixing Common Issues — the desktop-scale version of the same diagnosis, for when you need to hand someone the short answer.
📚 Recommended Resources
Disclosure: some links below are Amazon affiliate links. If you buy through them, I earn a small commission — at no extra cost to you — which helps keep this site running.
Operating Systems: Three Easy Pieces — free, and the right first stop
Remzi & Andrea Arpaci-Dusseau · Book, free online
Before any performance book, the virtualization section of OSTEP is where “why is the page cache full” stops being a puzzle. It builds address spaces, paging, TLBs, swapping, and replacement policy from first principles, so that the reclaim behaviour behind MemAvailable reads as a consequence rather than a quirk. The complete book is free at ostep.org — start there. There is no verified Amazon listing for it, so there is no affiliate link here, which is fine: the free version is the canonical one.
Systems Performance, 2nd ed.
Brendan Gregg · Addison-Wesley, 2020 · Book
The reference edition for exactly the discipline this article is arguing for: choose the metric that answers the question, then check utilization, saturation, and errors for each resource rather than reading whatever your agent happens to export. Its memory chapter covers the page cache, reclaim, swapping, and the /proc fields directly, and its methodology chapters are what turn “the box had free memory” into a repeatable diagnosis. If you buy one book off this page, this is it.
BPF Performance Tools
Brendan Gregg · Addison-Wesley, 2019 · Book
The companion volume, and the answer to the question this article deliberately stops short of: what do you do when the counters tell you that something is stalling but not which code path. It is a working catalogue of bcc and bpftrace tools, including memory and page-fault tracing, for when /proc and PSI have taken you as far as aggregate numbers can go.
The Linux Programming Interface
Michael Kerrisk · No Starch Press, 2010 · Book
The systems-programming side of the same subject, from the maintainer of the Linux man pages. Its treatment of virtual memory, mmap, shared memory, and mlock is what makes the difference between reclaimable and pinned memory concrete at the API level — which is precisely the distinction the naive free + cached arithmetic gets wrong. Older than the cgroup v2 material covered here, and still the clearest explanation of the syscall layer underneath it.