Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.

Linux memory problems are not diagnosed by one “used RAM” number or one command. Start by deciding which layer is failing: the host VM, a process, a cgroup or container, kernel allocations, or memory-pressure latency. Capture evidence before restarting anything, then follow the signal to the tool that can explain it.

The practical rule is to measure available memory, reclaim and swap activity, PSI stall time, allocation failures, and workload impact—not merely how full RAM appears.

A 10-minute, evidence-preserving triage

Run these commands before killing a process, dropping caches, or changing limits:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
date -Is
uname -a
cat /etc/os-release
free -h
cat /proc/meminfo
vmstat 1 10
cat /proc/pressure/memory
swapon --show
journalctl -k -b --no-pager | tail -n 200

free is only a summary. /proc and VM interfaces provide the counters needed to explain it. vmstat shows whether reclaim and swap are active, while PSI shows whether tasks are actually being delayed. Preserve these outputs and kernel logs before a restart destroys the timeline.

What “memory usage” means

  • MemAvailable is generally more useful than MemFree: it estimates memory that can be allocated without swapping, although the estimate depends on kernel policy and workload.
  • Anonymous memory holds process heaps, stacks and other private pages. It is usually less directly reclaimable than clean file cache.
  • Page cache caches files and can be reclaimed under pressure. High cache is not automatically a leak.
  • Slab stores kernel objects. SReclaimable can often be reclaimed; SUnreclaim generally cannot be reclaimed in the same way.
  • SwapUsed does not prove a fault. Look for active swap-in/swap-out, major faults, PSI and latency.
  • Committed virtual memory is an allocation promise, not resident RAM.
  • RSS counts resident mapped pages and can count shared pages in several processes. Use PSS when attributing shared memory.
  • Virtual size can grow while RSS stays flat; that is address-space reservation, not necessarily physical consumption.
  • Cgroup memory is charged according to the hierarchy and includes categories such as user memory, cache, kernel data and TCP buffers.

A machine can have free RAM yet fail a high-order allocation because of fragmentation, NUMA locality, a cgroup limit or a device-specific constraint. Conversely, a machine using nearly all RAM may be healthy if reclaim is cheap and PSI is near zero.

Find a growing process or service

pid=1234
ps -o pid,ppid,comm,%mem,rss,vsz,stat -p "$pid"
cat /proc/$pid/status
cat /proc/$pid/smaps_rollup 2>/dev/null
pmap -x "$pid" 2>/dev/null
pidstat -r -p "$pid" 1

Compare samples over time. Increasing RssAnon or PSS suggests private or shared resident growth. Increasing virtual size with stable RSS usually indicates reservations or mappings. File-backed mappings may be mapped files or libraries, while shared memory can be IPC, tmpfs or graphics buffers.

Do not call every increase a leak. Distinguish:

  • Language-level leak: live application objects remain reachable.
  • Allocator retention: freed objects stay in arenas for reuse.
  • Fragmentation: free space exists but cannot be returned or used efficiently.
  • Kernel-accounted memory: page tables, sockets, slab or other charges grow outside the application heap.

Use the appropriate application tool: Valgrind or Massif for controlled reproductions, AddressSanitizer/LeakSanitizer for instrumented builds, heaptrack, and runtime profilers for Java, Go, Python, Rust or other managed environments. Kernel tools cannot prove a user-space object leak.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

For a systemd service:

systemctl show example.service 
  -p MemoryCurrent -p MemoryPeak -p MemoryHigh -p MemoryMax 
  -p ManagedOOMMemoryPressure -p ManagedOOMSwap

Separate global OOM from cgroup OOM

journalctl -k -b --no-pager | grep -iE 
  'out of memory|oom-kill|killed process|memory cgroup'
dmesg -T | grep -iE 
  'out of memory|oom-kill|killed process|memory cgroup'

A global OOM occurs after the system cannot satisfy an allocation through reclaim and other handling. A memcg OOM occurs when a cgroup reaches its boundary; the host can still have substantial available memory. The selected victim is influenced by cgroup scope, allocation context, badness scoring and oom_score_adj, so the killed process is not always the root cause.

Container and cgroup v2 checks

cat /proc/$pid/cgroup
cg=/sys/fs/cgroup/example.slice/example.service
cat "$cg/memory.current"
cat "$cg/memory.peak"
cat "$cg/memory.high"
cat "$cg/memory.max"
cat "$cg/memory.events"
cat "$cg/memory.events.local"
cat "$cg/memory.stat"
cat "$cg/memory.pressure" 2>/dev/null

These files are documented in the cgroup v2 memory controller. memory.high is a reclaim and throttling boundary, not a direct OOM trigger. memory.max is the hard limit. memory.events is hierarchical; use memory.events.local when you need events generated in that cgroup itself. Counters are cumulative, so record two samples.

Interpret events as follows:

  • high: the group crossed its high boundary and tasks entered direct reclaim.
  • max: usage approached or exceeded the hard boundary.
  • oom: an allocation could not be satisfied within the limit.
  • oom_kill and oom_group_kill: processes were killed by cgroup OOM handling.

In Kubernetes, an OOMKilled container is not proof of a node-wide OOM. Check the pod’s memory limit, QoS class, node allocatable memory, kubelet eviction thresholds, sidecars and the runtime’s cgroup path. Page cache and some kernel categories can also be charged to the container.

PSI: measure harm, not just consumption

cat /proc/pressure/memory
cat "$cg/memory.pressure"

Pressure Stall Information reports time tasks were unable to make progress because of memory pressure. some means at least some tasks stalled; full means all non-idle tasks stalled during measured periods. avg10, avg60 and avg300 are rolling averages; total is cumulative microseconds. PSI answers “is the workload suffering?” while meminfo and cgroup statistics answer “what is being consumed?”

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Swap, reclaim and latency

vmstat 1
sar -W 1
cat /proc/meminfo
swapon --show
iostat -xz 1

Background reclaim by kswapd is normal. Direct reclaim runs in the allocating task and can add immediate latency. Rising si/so, major faults, disk saturation and PSI together indicate harmful pressure more convincingly than swap presence alone. Changing vm.swappiness or disabling swap without measurements can turn a recoverable burst into an OOM.

Dropping caches is a diagnostic experiment, not a repair:

sync
echo 3 | sudo tee /proc/sys/vm/drop_caches

It can evict useful data, create a performance cliff and leave the actual leak or limit untouched. Never use it as a routine production fix.

Slab and kernel-memory growth

cat /proc/slabinfo
slabtop -o
grep -E 'Slab|SReclaimable|SUnreclaim|KernelStack|PageTables|Percpu|Vmalloc' /proc/meminfo

Large dentry/inode caches may be legitimate; persistent unreclaimable growth can indicate sockets, page tables, kernel stacks, a driver or a subsystem leak. Allocation tracepoints are useful when available:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
mount -t tracefs nodev /sys/kernel/tracing 2>/dev/null || true
cd /sys/kernel/tracing
echo 1 > events/kmem/kmalloc/enable
echo 1 > events/kmem/kfree/enable
cat trace_pipe

Event names and fields depend on kernel configuration. See the kmem tracepoint documentation. Correlate allocation and free activity with a controlled workload rather than treating a busy call site as proof of a leak.

Kernel leaks: kmemleak, page owner and SLUB

kmemleak requires CONFIG_DEBUG_KMEMLEAK and debugfs. It tracks allocations such as kmalloc, vmalloc and slab allocations, then scans for references:

mount -t debugfs nodev /sys/kernel/debug 2>/dev/null || true
cat /sys/kernel/debug/kmemleak
echo clear > /sys/kernel/debug/kmemleak
# Reproduce the suspected path
echo scan > /sys/kernel/debug/kmemleak
cat /sys/kernel/debug/kmemleak

Output is a list of possible leaks, not proof. False positives and negatives exist; page allocations and ioremap allocations are not tracked. Scanning and instrumentation add overhead, and the documented automatic scan interval is 600 seconds unless changed.

page owner answers a different question: which allocation stack owns physical pages? It is disabled by default and normally enabled at boot with page_owner=on. Inspecting /sys/kernel/debug/page_owner can reveal page hogs and fragmentation, but enabling it changes memory overhead and should preferably be tested on a reproduction system.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Rank #4
Sale
Linux Device Drivers, 3rd Edition
  • Used Book in Good Condition

For slab corruption or consistency problems, boot-time slub_debug options such as F (sanity checks), Z (red zones), P (poisoning) and U (user tracking) may help. Exact combinations vary by kernel; debugging changes timing and memory use.

Fragmentation, NUMA and huge pages

numactl --hardware
numastat -m
cat /proc/buddyinfo
cat /proc/pagetypeinfo
grep -iE 'Huge|AnonHuge|ShmemHuge' /proc/meminfo
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

Global free memory does not guarantee a contiguous high-order block. NUMA policy, physical fragmentation, CMA reservations, huge-page availability and device constraints can all cause allocation failure. Do not disable transparent huge pages or alter compaction settings as a generic cure: the change can affect TLB efficiency, page-fault cost, footprint and latency in opposite directions for different workloads.

Tracing and postmortem analysis

perf stat -p "$pid" -e page-faults,major-faults,minor-faults,context-switches sleep 30
sudo perf record -a -g -- sleep 30
sudo perf report

perf, ftrace/tracefs, eBPF tools such as bpftrace or BCC, drgn, DAMON and trace-cmd can connect stalls or allocations to code paths. Access may require capabilities such as CAP_PERFMON, suitable perf_event_paranoid, BTF, kernel configuration and distribution packages; consult perf security guidance.

For crashes or hangs, prepare kdump/crash in advance, preserve the ftrace ring buffer and retain the exact kernel build, modules, command line, matching vmlinux and debug symbols. The kernel tracing guide describes retaining events before an oops. A dump without matching symbols may provide only an address, not an actionable allocation or reclaim path.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Safe mitigation order

  1. Contain: shed optional workload, pause a rollout, or stop a known runaway job if service survival requires it.
  2. Preserve evidence: save kernel logs, cgroup counters, PSI, meminfo and process maps before restarting.
  3. Apply a bounded control: use an appropriate cgroup limit, workload concurrency limit or temporary service isolation.
  4. Fix the cause: repair the application leak, allocator behavior, kernel subsystem, NUMA placement or sizing error.

Avoid blindly killing the largest RSS process, increasing every memory limit, changing swappiness, disabling THP or dropping caches. Each can hide the symptom while increasing the next failure’s blast radius.

systemd-oomd

systemctl status systemd-oomd
oomctl dump
journalctl -u systemd-oomd --no-pager
systemctl show example.service -p ManagedOOMMemoryPressure -p ManagedOOMSwap

systemd-oomd is a userspace policy mechanism using PSI and cgroup v2; it is not the kernel OOM killer. Unified cgroups, memory accounting, PSI support and eligible service hierarchy are required for the expected behavior. Swap-disabled or incomplete cgroup setups can change its effectiveness.

Quick playbook

Symptom First evidence Follow-up
Host is slow vmstat, PSI, meminfo, iostat perf, ftrace, NUMA and cgroup checks
One process grows status, smaps_rollup, PSS over time Runtime profiler, ASan/LSan or heaptrack
Container is OOMKilled memory.current/max/events, kernel log Hierarchy, kubelet and runtime events
Slab grows slabtop, slabinfo, meminfo kmem tracepoints, page owner, vendor analysis
Kernel allocation fails buddyinfo, pagetypeinfo, NUMA and huge-page counters Compaction tracing and page-owner stacks
System freezes before logging OOM PSI, persistent logs and preconfigured tracing kdump/crash with matching symbols

Optional observability services

Native Linux tools should be the first step. Hosted platforms can add history and fleet correlation: Grafana Cloud offers metrics, logs, profiles and Kubernetes monitoring; its displayed plans are usage-based, so ingestion, retention and host hours determine the bill. Datadog and New Relic can correlate memory with application telemetry, while Red Hat subscriptions provide distribution-specific support and escalation. None replaces page-owner, kmemleak or local cgroup evidence when the failure is kernel-internal.

Frequently Asked Questions

Does high Linux memory usage mean a leak?

No. Linux intentionally uses idle RAM for page cache and kernel objects. Check MemAvailable, reclaim, PSI, swap activity and latency, then measure growth by process or cgroup over time.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Can a container be OOM-killed while the host has free RAM?

Yes. A cgroup can hit memory.max or another hierarchy constraint independently of host-wide availability. Inspect memory.current, memory.max, memory.events and the container’s cgroup path.

Is kmemleak proof of a kernel leak?

No. kmemleak reports possible unreferenced objects and has documented false positives and false negatives. Confirm with controlled reproduction and allocation or page-ownership evidence.

The Bottom Line

Diagnose Linux memory in layers: establish pressure and impact, identify the process or cgroup, separate reclaimable cache from unreclaimable growth, and only then use tracing, kmemleak, page owner or profilers. Preserve evidence before applying a mitigation, because the fastest-looking fix is often the one that removes the clues.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.