Linux Performance Analysis

May 31, 2026 29 min read Systems Administration

linuxperformanceprofilingbpftraceperfflamegraphsysadminobservability

Contents

Performance analysis on Linux is one of those disciplines where the tools are abundant, the documentation is scattered, and the temptation to reach for the nearest command and start poking is nearly irresistible. That instinct is expensive. A production system under load is not the place to run random commands and hope for insight. What you need is a methodology — a way to systematically eliminate hypotheses and converge on a root cause before your incident window closes.

This post is a reference for working performance engineers and SREs. It covers the USE method as a structured approach, the classic first-60-seconds checklist, CPU and memory analysis, disk and network I/O, dynamic tracing with bpftrace, and syscall inspection with strace. It is written to be dense. If you want theory at the expense of commands, look elsewhere.

The USE Method: A Checklist, Not a Fishing Expedition

Brendan Gregg coined the USE method to bring systematic thinking to performance analysis. For every resource in the system, you ask three questions:

Utilization — What fraction of time is this resource busy? (e.g., CPU at 90%)
Saturation — Is work piling up waiting for this resource? (e.g., run queue length > CPU count)
Errors — Are there error counters incrementing? (e.g., NIC rx_errors, disk I/O errors)

The insight is that saturation is often more important than utilization. A disk at 70% utilization with an await latency of 200ms is saturated — queued requests are waiting far longer than the hardware should take. A CPU at 80% utilization with a run queue of 1 is not saturated. These are very different situations, and utilization alone cannot distinguish them.

The method also forces you to work through a resource checklist rather than jumping straight to the resource you suspect. The following resources should be evaluated for each of U, S, and E:

USE Checklist
=============

Resource         | Utilization        | Saturation              | Errors
-----------------|--------------------|-------------------------|-------------------------
CPUs             | mpstat %usr+%sys   | vmstat r > #CPUs        | perf stat errors
Memory           | free -m %used      | vmstat si/so > 0        | dmesg OOM kills
Disk (each)      | iostat %util       | iostat await spike      | dmesg, /sys/..../errors
Network (each)   | sar -n DEV %ifutil | ifconfig overruns       | ethtool -S rx_errors
File descriptors | /proc/sys/fs/file  | —                       | dmesg EMFILE

The Brendan Gregg Linux performance observability tools map places each tool in context across the system stack. Understanding which layer a tool observes is critical — a tool measuring at the VFS layer will not tell you about hardware firmware latency, and a tool looking at hardware counters will not tell you about application-level lock contention.

Linux Performance Observability Tools Map
==========================================

  Applications         top, htop, pidstat, time, perf record -g
  ----------------------
  System Libraries     ltrace, strace (partial), perf probe
  ----------------------
  System Calls         strace, perf trace, bpftrace tracepoint:syscalls
  ----------------------
  VFS                  bpftrace kprobe:vfs_*, opensnoop, filesnoop
  ----------------------
  File Systems         df, du, inotifywait, ext4slower (BCC)
  ----------------------
  Block Layer          iostat, iotop, blktrace, biolatency (BCC)
  ----------------------
  Device Drivers       /sys/block/*/stat, blktrace D events
  ----------------------
  Hardware (Storage)   smartctl, nvme-cli, hdparm
  ----------------------
  Network Stack        ss, netstat, nstat, tcpconnect (BCC)
  ----------------------
  NIC Driver/Hardware  ethtool -S, /sys/class/net/*/statistics
  ----------------------
  CPU / Memory HW      perf stat (PMU counters), turbostat, pcm

Every tool in that map answers different questions. Working top-down from application symptoms toward hardware is the most efficient path in most incidents.

The First 60 Seconds on a Slow System

Before reaching for anything specialized, run through this ten-command checklist in order. Each command feeds context into the next, and together they give you a full-system overview in about a minute.

1. uptime

$ uptime
 14:32:07 up 42 days,  3:12,  2 users,  load average: 8.42, 6.71, 4.30

Load averages are the 1-minute, 5-minute, and 15-minute exponential moving averages of runnable plus uninterruptible-sleep (D-state) task counts. On an 8-CPU system, a 1-minute load of 8.42 means the system is at or slightly above saturation right now. The trend matters: a load of 8.42/6.71/4.30 is rising — something recently got worse. A load of 4.30/6.71/8.42 is recovering.

Important caveat: load average includes processes in D state (waiting for I/O), not just CPU-bound processes. A load of 16 on an 8-CPU system could be 16 I/O-blocked processes and 0% CPU utilization. Always cross-reference with CPU and I/O metrics.

2. dmesg | tail

$ dmesg | tail -20

Look for: OOM kill messages (Out of memory: Kill process), hardware errors (disk errors, NIC errors, PCIe errors), kernel warnings (BUG:, WARNING:), file system errors. If anything appears here, it escalates to the top of your investigation.

3. vmstat 1

$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 4  2  51200  82304  12288 921600    0    0   480   320 4521 8103 28  8 52 12  0
 6  1  51200  79872  12288 921600    0    0   512   288 4812 8441 31  9 47 13  0
 3  0  51200  81920  12288 921600    0    0   256   192 4103 7820 26  7 60  7  0

vmstat 1 — annotated output
============================

Column  | Meaning
--------|----------------------------------------------------------
r       | Runnable tasks (running + waiting for CPU)
        | If r > nCPU consistently: CPU saturation
b       | Blocked in uninterruptible sleep (D state, usually I/O)
        | Elevated b: disk or network I/O blocking processes
swpd    | Amount of virtual memory swapped (kB)
        | Non-zero is a yellow flag; growing is a red flag
free    | Idle memory (kB) — ignore this number; see 'available'
buff    | Kernel buffer cache (metadata) — reclaimable
cache   | Page cache (file data) — reclaimable
si      | Swap-in: pages read from swap per second
so      | Swap-out: pages written to swap per second
        | Any si/so > 0 under normal load = memory pressure
bi      | Blocks read from block devices per second (reads)
bo      | Blocks written to block devices per second (writes)
in      | Interrupts per second
cs      | Context switches per second — high cs + high sy = kernel overhead
us      | User-space CPU time %
sy      | Kernel CPU time % — high sy warrants investigation
id      | Idle CPU %
wa      | CPU time waiting for I/O — high wa = I/O bottleneck
st      | CPU time stolen by hypervisor — any non-zero warrants attention

In the example above: r=4-6 on what might be a 4-CPU system is borderline saturation. wa=12-13% means the CPUs are blocking on I/O 12% of the time. b=1-2 confirms processes are stuck in D state. This points toward a disk I/O investigation.

4. mpstat -P ALL 1

$ mpstat -P ALL 1
Linux 6.8.0-117-generic   05/31/26   _x86_64_   (8 CPU)

14:32:10  CPU   %usr  %nice  %sys %iowait  %irq  %soft %steal %idle
14:32:11  all  28.41   0.00  8.12   12.33  0.12   0.44   0.00  50.58
14:32:11    0  31.00   0.00  9.00   14.00  0.00   1.00   0.00  45.00
14:32:11    1  29.00   0.00  8.00   11.00  0.00   0.00   0.00  52.00
14:32:11    2  14.00   0.00  5.00   24.00  0.00   0.00   0.00  57.00
14:32:11    3  39.00   0.00 11.00    8.00  0.00   0.00   0.00  42.00

Per-CPU breakdown lets you detect imbalance. A single CPU at 100% %usr while others are idle is not a system-wide CPU problem — it is a single-threaded application bottleneck. Uneven %iowait across CPUs can indicate an IRQ affinity issue where all I/O interrupts land on one CPU. %steal > 0 means the hypervisor is reclaiming CPU cycles; escalate to your cloud provider if this is persistent.

5. pidstat 1

$ pidstat 1
14:32:12  UID   PID %usr %system %guest %wait   %CPU  CPU  Command
14:32:13    0  1234 42.0    3.00   0.00   1.00  45.00    2  java
14:32:13  500  5678  8.0    1.00   0.00  15.00   9.00    0  postgres
14:32:13    0  9012  0.0   12.00   0.00   0.00  12.00    7  kworker

The %wait column (CPU scheduler wait time, not I/O wait) is valuable for diagnosing CPU saturation — a process with high %wait is being starved by the scheduler. pidstat -d 1 shows per-process disk I/O, useful to identify which process is generating the I/O you see in vmstat.

6. iostat -xz 1

$ iostat -xz 1
Device    r/s    w/s  rMB/s  wMB/s r_await w_await aqu-sz  %util
sda      48.0   32.0   1.80   2.40   22.40   18.60   1.83  72.00
nvme0n1  120.0  85.0  14.20  18.80    0.18    0.22   0.04   8.40

iostat -xz 1 — annotated output
================================

Column   | Meaning
---------|------------------------------------------------------------
r/s      | Read requests per second (IOPS reads)
w/s      | Write requests per second (IOPS writes)
rMB/s    | Read throughput in MB/s
wMB/s    | Write throughput in MB/s
r_await  | Average read latency in milliseconds — KEY METRIC
w_await  | Average write latency in milliseconds — KEY METRIC
await    | Average combined I/O latency (older iostat versions)
aqu-sz   | Average queue size — > 1 means requests are queuing
%util    | Percentage of time the device had outstanding I/O
         | Misleading for SSDs: an NVMe can be "saturated" at 20% util

Latency baselines:
  HDD:       5–15 ms normal, > 30 ms problematic
  SATA SSD:  0.1–0.3 ms normal, > 1 ms problematic
  NVMe SSD:  0.02–0.05 ms normal, > 0.5 ms problematic

In the example, sda at r_await=22.4ms is near the high end for a spinning disk, and aqu-sz=1.83 confirms requests are queuing — this disk is struggling. The NVMe is healthy at r_await=0.18ms. The -z flag suppresses idle devices, keeping output clean.

7. free -m

$ free -m
              total   used   free  shared  buff/cache  available
Mem:          15953  12041    804     287        3108       3245
Swap:          4095   1024   3071

The available column is the number you care about. Linux fills free memory with page cache, so free is normally small and that is correct behavior. available is the estimated memory that can be given to a process immediately, accounting for reclaimable cache. Here: 3245 MB available, 1024 MB of swap in use — memory is tight but not yet in crisis. If swap usage is growing and available is falling toward zero, you are approaching OOM territory.

8. sar -n DEV 1

$ sar -n DEV 1
14:32:15   IFACE  rxpck/s  txpck/s  rxkB/s  txkB/s  rxcmp/s  txcmp/s  %ifutil
14:32:16   eth0   1420.0   1380.0  9840.0  8920.0     0.0      0.0      0.95
14:32:16   lo      12.0     12.0    1.2     1.2       0.0      0.0      0.00

%ifutil is NIC saturation. Approaching 100% means the NIC is a bottleneck. rxkB/s and txkB/s tell you the actual throughput — compare against your NIC’s rated bandwidth (use ethtool eth0 to check link speed).

9. sar -n TCP,ETCP 1

$ sar -n TCP,ETCP 1
14:32:16   active/s  passive/s  iseg/s  oseg/s
14:32:17     12.00      8.00   1840.0  1760.0

14:32:16   atmptf/s  estres/s  retrans/s  isegerr/s  orsts/s
14:32:17      0.00      0.00      14.00      0.00      2.00

retrans/s of 14 per second is elevated and worth investigating — retransmits indicate packet loss somewhere in the network path. estres/s (established resets) and orsts/s (outbound resets) indicate connections being abnormally terminated.

10. top

$ top

Press P to sort by CPU, M to sort by memory. Look at the summary line: load average, tasks, CPU summary, memory. Then scan the process list for the obvious outliers. top is not precise enough for deep analysis, but it gives orientation. After running through the first nine commands, you should already have a strong hypothesis by the time you open top.

CPU Performance Analysis

Utilization vs. Saturation vs. Steal

High CPU utilization is not inherently a problem. A system running at 85% %usr is doing useful work. The questions are: is it the right work, is it efficient, and is any work being delayed?

%usr (user space) — application CPU. High is expected under load.
%sys (kernel) — system calls, scheduling, network/IO stack. Persistently above 20-30% without obvious cause warrants investigation.
%iowait — the CPU is idle but at least one process is waiting for I/O. This is not CPU saturation; it is I/O saturation expressed in CPU terms.
%steal — the hypervisor took cycles your VM thought it owned. Even 5-10% steal significantly degrades latency-sensitive workloads.
r column in vmstat — the run queue. If r consistently exceeds the number of CPUs, tasks are waiting to run. This is CPU saturation, not just high utilization.

Hardware Performance Counters with perf stat

$ perf stat -a sleep 5

 Performance counter stats for 'system wide':

     19,543,210,432  cycles                    #    3.908 GHz
     12,847,330,210  instructions              #    0.66  insn per cycle
      1,240,512,834  cache-references
        412,301,920  cache-misses              #   33.24% of all cache refs
        290,441,100  branch-instructions
          8,201,440  branch-misses             #    2.82% of all branches

       5.001234567 seconds time elapsed

IPC (instructions per cycle) is the most important derived metric here. An IPC of 0.66 is low — modern CPUs can execute 2-4 instructions per cycle when the workload is cache-friendly. Low IPC (below 1.0) combined with high cache-misses tells you the CPU is stalling waiting for data from RAM — a memory-access bound workload. Low IPC with low cache misses but high branch-misses points to branch misprediction. These are fundamentally different optimization targets.

Install perf on Ubuntu/Debian:

sudo apt install linux-tools-$(uname -r) linux-tools-generic

perf top and perf record

perf top gives a live, top-like view of where CPU time is being spent, broken down by function symbol:

$ sudo perf top
   Overhead  Shared Object     Symbol
     23.41%  [kernel]          [k] __schedule
     18.32%  libc.so.6         [.] malloc
     12.10%  myapp             [.] process_request
      8.44%  [kernel]          [k] copy_user_enhanced_fast_string

Symbols appearing in the kernel ([k]) vs userspace ([.]) tell you where time is spent. High __schedule in the kernel often correlates with high context switch rates.

For deeper analysis, record a call-graph profile:

$ sudo perf record -F 99 -g -p <pid> sleep 30

Flags: -F 99 samples at 99 Hz (prime number avoids lock-step with 100 Hz timer events), -g captures call graphs (stack traces), -p <pid> targets a specific process. The output is written to perf.data.

Explore the recorded data:

$ sudo perf report

This opens an interactive TUI. Navigate with arrow keys, press Enter to expand a symbol and see its callers. The % column shows what fraction of samples included that function anywhere in the call stack. Look for hot leaves (functions at the bottom of the call tree that appear in many samples) — these are where the CPU is actually spending time.

Flamegraphs

Flamegraphs are the most effective visualization for call-stack profiling data. The axes:

Flamegraph Anatomy
==================

  ^
  |   [  malloc  ]
  |   [ process_request          ]
  |   [ handle_connection                              ]
  |   [ epoll_wait ][ accept4 ][ writev ]
  |   [ __libc_start_main                                        ]
  +-------------------------------------------------------------> x-axis

  Y-axis: call stack depth (bottom = entry point, top = leaf function)
  X-axis: number of samples (width = proportion of total CPU time)

  Wide frames  = hot code paths — this is where time is spent
  Narrow frames = called infrequently
  Tall towers  = deep call chains, not necessarily expensive
  Plateaus     = flat top means a leaf function — the actual work

  How to read:
  - Find the widest frame near the top (leaf level)
  - That function is consuming the most CPU
  - Read downward to understand its callers
  - Unexpected callers at the same level = surprise hot paths

Generate a flamegraph from perf.data using Brendan Gregg’s FlameGraph scripts (https://github.com/brendangregg/FlameGraph):

$ git clone https://github.com/brendangregg/FlameGraph
$ sudo perf script | ./FlameGraph/stackcollapse-perf.pl | \
    ./FlameGraph/flamegraph.pl > flame.svg

Open flame.svg in a browser. Click frames to zoom in. Search by function name with Ctrl+F. The SVG is interactive.

One practical tip: use --no-inline with perf record if your application has aggressive inlining — otherwise inlined functions disappear from the profile and their callers look hotter than they are.

CPU Frequency Scaling and Thermal Throttling

A workload that looks CPU-bound but has mysteriously low throughput may be thermally throttled. Check:

$ cpupower frequency-info
$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
$ turbostat --interval 1

When CPUs throttle, clock frequency drops and IPC from perf stat will appear to decline even without code changes. turbostat shows per-core frequencies, C-state residency, and temperature (on supported hardware). If frequency is oscillating or stuck well below nominal, cooling is your problem.

Memory Performance Analysis

What Free Memory Means on Linux

Linux uses all available RAM as page cache. This is correct behavior, not a memory leak. The kernel will evict cache pages under memory pressure to make room for application allocations. The confusion arises because tools like free show a small free column and naive observers panic.

$ free -m
              total   used   free  shared  buff/cache  available
Mem:          31906  18432   2048    1024       11426      12800

Here: free=2048 MB but available=12800 MB. The 11 GB of buff/cache is reclaimable. There is no memory pressure at all. The column to monitor is available.

/proc/meminfo Key Fields

$ cat /proc/meminfo
MemTotal:       32672416 kB
MemFree:         2097152 kB
MemAvailable:   13107200 kB
Buffers:          126976 kB
Cached:         11534336 kB
SwapCached:        32768 kB
Active(file):    6815744 kB
Inactive(file):  4718592 kB
Dirty:            524288 kB
Writeback:          8192 kB
AnonPages:       7340032 kB
Mapped:          1048576 kB
Shmem:           1048576 kB
KReclaimable:    1572864 kB

Field breakdown:

Field	What it tells you
`MemAvailable`	Estimated available memory — the real free memory number
`Buffers`	Block device metadata cache — small, reclaimable
`Cached`	Page cache (file data) — reclaimable
`SwapCached`	Pages in both RAM and swap — being brought back from swap
`Active(file)`	Recently used page cache — will be evicted last
`Inactive(file)`	Older page cache — first to be evicted under pressure
`Dirty`	Modified pages not yet written to disk — flush in progress
`Writeback`	Pages currently being written to disk
`AnonPages`	Process heap/stack — cannot be evicted without swapping
`Mapped`	Memory-mapped files (mmap’d executables, shared libs)
`Shmem`	Shared memory, including tmpfs
`KReclaimable`	Kernel allocations that can be reclaimed under pressure

Dirty is worth monitoring: persistently high Dirty (several GB) means the system is accumulating writes faster than it can flush them. When the kernel finally flushes, you will see bo spike in vmstat and w_await climb in iostat. Tune with vm.dirty_ratio (hard limit, triggers synchronous writes) and vm.dirty_background_ratio (soft limit, triggers background flushing):

$ sysctl vm.dirty_ratio vm.dirty_background_ratio
vm.dirty_ratio = 20
vm.dirty_background_ratio = 10

Swap and Memory Pressure

Swap-in (si) and swap-out (so) in vmstat are the primary swap indicators. Any non-zero value under steady-state load is a warning. Swap activity means the kernel is evicting anonymous pages (heap/stack), which is expensive — anonymous pages must be serialized to the swap device before the memory can be reused, and faulted back in on access.

$ swapon --show
NAME      TYPE  SIZE USED PRIO
/dev/sda3 partition  4G  1.2G   -2

For per-process memory accounting, smem provides accurate PSS (Proportional Set Size), which divides shared pages fairly among processes rather than counting them fully for each:

$ smem -t -k -s pss
  PID User     Command                     Swap      USS      PSS      RSS
 1234 appuser  java -Xmx8g -jar app.jar    128M    6.8G     7.1G     7.4G
 5678 postgres postgres: worker            512K   48.2M    62.4M    74.1M

USS (Unique Set Size) is private memory, PSS is fair-share of shared, RSS is the traditional resident set (overcounts shared pages). PSS is the most accurate for capacity planning.

OOM Killer

When MemAvailable reaches zero and the kernel cannot reclaim enough memory, the OOM (Out of Memory) killer selects a process and kills it. Its selection algorithm is based on oom_score, which roughly correlates with memory usage and process importance. Inspect OOM events:

$ dmesg | grep -i "oom\|killed process\|out of memory"
[842341.112233] Out of memory: Kill process 12345 (java) score 812 or sacrifice child
[842341.112890] Killed process 12345 (java) total-vm:16777216kB, anon-rss:8388608kB

To protect a critical process from OOM kills (e.g., your monitoring agent):

$ echo -1000 > /proc/<pid>/oom_score_adj

Valid range is -1000 (never kill) to 1000 (kill first). Setting a sensitive process to -1000 means it will be the last killed — the kernel will destroy everything else first.

Pressure Stall Information (PSI)

PSI is available in kernels 4.20+ and is enabled by default on Ubuntu 22.04+, Debian 12, and most modern distributions (check: cat /proc/pressure/memory). If the file exists, PSI is active. If not, add psi=1 to your kernel command line.

$ cat /proc/pressure/memory
some avg10=0.42 avg60=0.18 avg300=0.05 total=12340567
full avg10=0.08 avg60=0.03 avg300=0.01 total=2341234

some = at least one task was stalled waiting for memory. full = all tasks were stalled (the CPU was idle because everything was waiting for memory). avg10, avg60, avg300 are exponential moving averages over 10s, 60s, 300s. A full avg10 above 5-10% indicates serious memory pressure. PSI is the most reliable signal for triggering automated memory remediation (e.g., killing caches, triggering OOM earlier) in container environments like cgroups v2.

Memory Leak Detection

For development: valgrind --leak-check=full ./myapp. For production processes you cannot restart: track AnonPages in /proc/meminfo and per-process VmRSS/VmPeak in /proc/<pid>/status over time. A monotonically growing AnonPages that never shrinks is a memory leak signature.

Disk I/O Performance Analysis

iostat -xz 1 Deep Dive

$ iostat -xz 1 5
Device     r/s    w/s  rkB/s  wkB/s r_await w_await  aqu-sz  %util
sda       48.3   31.7  1843.2 2410.4   22.4   18.6    1.83   72.1
sdb        0.2    0.5    12.8   48.3    4.2    3.8    0.00    0.3
nvme0n1  234.0  180.0 28672.0 24576.0   0.18   0.22   0.04    8.4

Read await and r_await/w_await first. These are your latency numbers. Then look at aqu-sz: a queue depth consistently above 1 means requests are piling up. %util for spinning disks correlates well with saturation; for NVMe, the device handles multiple queues internally and %util of 8.4% at 414 IOPS with 0.18ms latency is completely healthy — the device is not saturated at all.

sda in this example: await=22ms is acceptable but elevated for an HDD. aqu-sz=1.83 means on average nearly two requests are queued — this disk is approaching saturation. Investigate what is generating 48+31=79 IOPS on it.

iotop

iotop shows real-time per-process I/O:

$ sudo iotop -o
Total DISK READ:       1.84 M/s | Total DISK WRITE:      2.41 M/s
TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
1234 be/4  appuser   1.62 M/s    0.00 B/s  0.00 %  42.31 %  java
5678 be/4  postgres  0.22 M/s    1.20 M/s  0.00 %  12.50 %  postgres: autovac

The -o flag shows only processes with active I/O, avoiding clutter. The IO% column (actual I/O wait fraction for that process) identifies the top consumers immediately.

blktrace for Low-Level Block I/O Tracing

When iostat tells you there is a problem but not why, blktrace reveals the full lifecycle of every I/O request:

$ sudo blktrace -d /dev/sda -o - | blkparse -i -
  8,0    3    1     0.000000000  1234  Q  WS 839624192 + 8 [java]
  8,0    3    2     0.000012340  1234  G  WS 839624192 + 8 [java]
  8,0    0    3     0.000015120     0  I  WS 839624192 + 8 []
  8,0    0    4     0.000021340  1234  D  WS 839624192 + 8 [java]
  8,0    0    5     0.022341200     0  C  WS 839624192 + 8 [0]

The event codes: Q=queued, G=get request (allocated from pool), I=inserted to scheduler queue, D=dispatched to driver, C=completed. The timestamp difference between D and C is the actual device service time (22ms in this example). The difference between Q and D is the scheduler queue time. If scheduler time is large, you may need to tune the I/O scheduler.

Check the current I/O scheduler:

$ cat /sys/block/sda/queue/scheduler
[mq-deadline] kyber none

$ cat /sys/block/nvme0n1/queue/scheduler
[none]

none (no scheduler, direct pass-through) is appropriate for NVMe and fast SSDs that manage their own queuing internally. mq-deadline is a good general-purpose choice for SATA SSDs and HDDs. kyber is designed for low-latency SSDs with multiple queue depths.

Benchmarking with fio

Never assume — measure. Use fio to characterize your storage’s actual capabilities:

# Sequential read: simulate database log reading
$ fio --name=seqread --rw=read --bs=1M --size=4G --numjobs=1 \
    --iodepth=32 --runtime=60 --group_reporting

# Sequential write: simulate log appending
$ fio --name=seqwrite --rw=write --bs=1M --size=4G --numjobs=1 \
    --iodepth=32 --runtime=60 --group_reporting

# Random 4K read: simulate database random access
$ fio --name=randread --rw=randread --bs=4k --size=4G --numjobs=4 \
    --iodepth=32 --runtime=60 --group_reporting

# Random 4K write: simulate database WAL and data files
$ fio --name=randwrite --rw=randwrite --bs=4k --size=4G --numjobs=4 \
    --iodepth=32 --runtime=60 --group_reporting

Key output: iops, bw (bandwidth), lat (latency percentiles — look at p99 and p99.9). If your production await numbers exceed the p99 latency from fio under similar load, the device itself is the bottleneck.

Network Performance Analysis

Socket Summary with ss

$ ss -s
Total: 1842
TCP:   1624 (estab 412, closed 1182, orphaned 18, timewait 1164)

High timewait counts (thousands) are normal for servers handling many short-lived HTTP connections — TIME_WAIT is the TCP protocol’s intentional 2*MSL wait period to ensure stale packets don’t collide with new connections. If TIME_WAIT is exhausting your local port range, tune:

$ sysctl net.ipv4.tcp_tw_reuse
$ sysctl net.ipv4.ip_local_port_range

close-wait is more concerning: a connection in CLOSE_WAIT means the remote side closed the connection but the local application has not called close() yet. A growing count of CLOSE_WAIT indicates an application bug — file descriptors leaking open connections.

NIC Statistics with ethtool

$ ethtool -S eth0 | grep -E 'errors|drop|miss|over'
     rx_errors: 0
     tx_errors: 0
     rx_dropped: 0
     tx_dropped: 14823
     rx_missed_errors: 0
     rx_over_errors: 0

tx_dropped of 14823 is serious — the NIC’s transmit queue is full and packets are being dropped before they leave the machine. This indicates NIC saturation or a qdisc (traffic control queue) configuration problem. Check:

$ tc -s qdisc show dev eth0
qdisc mq 0: root
 Sent 924311040 bytes 742842 pkt (dropped 14823, overlimits 0 requeues 18)

Drops at the qdisc layer confirm the transmit path is saturated. Mitigations include increasing txqueuelen (ip link set eth0 txqueuelen 10000) or increasing NIC transmit ring buffer size (ethtool -G eth0 tx 4096).

TCP Retransmit Monitoring

$ nstat -az | grep -i retrans
TcpRetransSegs               1247               0.0
TcpExtTCPSynRetrans           18               0.0
TcpExtTCPFastRetrans         892               0.0
TcpExtTCPSlowStartRetrans     42               0.0

TcpRetransSegs is the total retransmit counter. Use nstat in a loop to get a rate. Per-connection retransmit counts:

$ ss -ti dst <remote_ip>

The retrans:X/Y field in ss -ti output shows (retransmits since last ACK / total retransmits) per socket.

bpftrace for Dynamic Tracing

bpftrace is a high-level tracing language built on eBPF that runs on Linux 4.9+ without kernel recompilation. It compiles D-language-like programs into eBPF bytecode, attaches them to kernel and userspace probes, and runs them in kernel context — safely, because the eBPF verifier rejects programs that could crash the kernel or loop infinitely.

Install on Ubuntu 24.04 / Debian 12:

$ sudo apt install bpftrace

Current version as of 2025 is 0.24.x. Verify: bpftrace --version.

A note on safety: kprobe-based probes add a tiny overhead to every invocation of the probed function. On a hot kernel path like tcp_sendmsg (called millions of times per second), extended tracing sessions add measurable overhead. Keep tracing sessions short on production, or use tracepoints (static probes, lower overhead) where available.

Essential One-Liners

Trace all file opens with process name:

$ sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
    printf("%s(%d) opened: %s\n", comm, pid, str(args->filename));
}'

Count write syscalls by process over 10 seconds:

$ sudo bpftrace -e 'tracepoint:syscalls:sys_enter_write { @[comm] = count(); }
    interval:s:10 { print(@); clear(@); }'

Histogram of read() return sizes (bytes actually read):

$ sudo bpftrace -e 'tracepoint:syscalls:sys_exit_read /retval > 0/ {
    @bytes = hist(retval);
}'

Count malloc calls by process (userspace probe):

$ sudo bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc {
    @allocs[comm] = count();
}'

Block I/O latency histogram (microseconds):

$ sudo bpftrace -e '
tracepoint:block:block_rq_issue { @start[args->dev, args->sector] = nsecs; }
tracepoint:block:block_rq_complete
/@start[args->dev, args->sector]/
{
    @usecs = hist((nsecs - @start[args->dev, args->sector]) / 1000);
    delete(@start[args->dev, args->sector]);
}'

Identify processes making new TCP connections:

$ sudo bpftrace -e 'kprobe:tcp_connect {
    printf("%s(%d) -> %s\n", comm, pid, ntop(((struct sock *)arg0)->__sk_common.skc_daddr));
}'

Slow syscall detector — find syscalls taking more than 10ms:

$ sudo bpftrace -e '
tracepoint:raw_syscalls:sys_enter { @start[tid] = nsecs; }
tracepoint:raw_syscalls:sys_exit
/@start[tid] && (nsecs - @start[tid]) > 10000000/
{
    printf("slow syscall: %s pid=%d latency=%dms\n",
           comm, pid, (nsecs - @start[tid]) / 1000000);
    delete(@start[tid]);
}'

BCC Tools

The BCC (BPF Compiler Collection) package ships dozens of production-ready tracing tools. Install on Ubuntu 24.04:

$ sudo apt install bpfcc-tools linux-headers-$(uname -r)

The tools are installed as /usr/sbin/<toolname>-bpfcc. Key tools:

Tool	What it does
`execsnoop-bpfcc`	Traces all process executions (exec syscalls) in real time
`opensnoop-bpfcc`	Traces all file opens with path and PID
`tcpconnect-bpfcc`	Traces outbound TCP connection attempts
`tcpaccept-bpfcc`	Traces inbound TCP connections accepted
`biolatency-bpfcc`	Block I/O latency histogram — replaces blktrace for most use cases
`biotop-bpfcc`	top-like I/O per process, using BPF
`runqlat-bpfcc`	Run queue latency histogram — definitive CPU saturation diagnosis
`runqlen-bpfcc`	Run queue length over time
`profile-bpfcc`	CPU profiler — cheaper alternative to `perf record` for hotspot finding
`cachestat-bpfcc`	Page cache hit/miss rates
`cachetop-bpfcc`	Per-process page cache statistics

runqlat deserves special mention. CPU utilization at 75% does not tell you whether tasks are waiting to run. runqlat measures the time each task spends in the CPU run queue before it gets scheduled onto a CPU:

$ sudo runqlat-bpfcc 10
Tracing run queue latency... Hit Ctrl-C to end.

     usecs               : count     distribution
         0 -> 1          : 8423     |****************************************|
         2 -> 3          : 3201     |***************                         |
         4 -> 7          : 891      |****                                    |
         8 -> 15         : 312      |*                                       |
        16 -> 31         : 143      |                                        |
        32 -> 63         : 87       |                                        |
        64 -> 127        : 41       |                                        |
       128 -> 255        : 18       |                                        |
       256 -> 511        : 4        |                                        |
      1024 -> 2047       : 1        |                                        |

A healthy system shows most tasks scheduled in 0-3 microseconds. Latency extending into the 256-511 microsecond range means tasks are waiting a quarter-millisecond to run — on a latency-sensitive service (e.g., a database, an HTTP API), that directly adds to request tail latency. This is CPU saturation, even if %util looks moderate.

strace and ltrace

strace: Syscall Tracing

strace intercepts every system call made by a process, printing arguments and return values. It is the right tool when you need to know what a process is actually asking the kernel to do, especially when documentation is wrong, source is unavailable, or behavior is mysterious.

Attach to a running process:

$ sudo strace -p <pid>

Get a syscall summary (count and time per syscall type):

$ sudo strace -c -p <pid>
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- --------
 42.31    0.421300         421      1000           futex
 18.44    0.183400          18     10000           epoll_wait
 12.22    0.121800          12     10000           read
  8.11    0.080900           8     10000           write
  7.30    0.072800          72      1000           recvfrom

This summary reveals that 42% of syscall time is spent in futex — this process is heavily contended on mutexes. That is a threading/locking problem, not an I/O problem.

Filter to specific syscall groups with timing:

$ sudo strace -T -e trace=network,read,write -p <pid>
recvfrom(5, "HTTP/1.1 200 OK\r\n"..., 4096, 0, NULL, NULL) = 1432 <0.000241>
write(3, "processed\n", 10)             = 10 <0.000008>
recvfrom(5, "", 4096, 0, NULL, NULL)    = 0 <0.000005>

The <0.000241> at the end of each line is the time spent in that syscall (seconds). Slow recvfrom calls indicate network latency. Slow write calls on a file descriptor usually indicate fsync contention.

Use cases where strace is the definitive tool:

Application failing with “No such file or directory” but logs don’t say which file: strace -e trace=openat -p <pid> to see every path it tries
Process stuck in D state: attach strace, it will show you the syscall it is blocked in
Unexpected network connections: strace -e trace=network -p <pid>
Slow startup: strace -T -tt ./myapp 2>&1 | grep -E "open|stat" to find which files take time to open

Overhead warning: strace uses ptrace, which stops the process for every syscall. On a process making 100,000 syscalls/second, this can slow it by 5-10x. Never attach strace to a high-throughput production process without a specific hypothesis and readiness to detach immediately.

perf trace: Lower-Overhead Alternative

$ sudo perf trace -p <pid>

perf trace provides similar syscall visibility using eBPF under the hood — no ptrace stop per syscall, dramatically lower overhead (typically 1-2% vs 500-1000% for strace). Prefer it for production use.

ltrace: Library Call Tracing

$ ltrace -c ./myapp
% time     seconds  usecs/call     calls      function
------ ----------- ----------- --------- --------------------
 31.22    0.312200         312      1000 SSL_read
 18.44    0.184400          18     10000 malloc
 12.11    0.121100          12     10000 free

ltrace intercepts calls to shared libraries — libc, libssl, libpthread, etc. It is useful when you have a binary you cannot instrument any other way and want to understand what library calls it is making. It carries similar overhead to strace (ptrace-based) so treat it the same way.

Performance Tooling Quick Reference

Symptom-to-Tool Mapping

Symptom	First tools	Follow-up
High load average, low `%util` CPU	`vmstat r` (run queue), `ps aux \| grep ' D '` (D-state)	`iotop`, `iostat await`, `runqlat-bpfcc`
High load average, high `%iowait`	`iostat -xz 1` for await + util	`iotop`, `blktrace`, `biolatency-bpfcc`
High CPU `%usr`, low IPC	`perf stat` (cache-misses, IPC)	Flamegraph, `perf report`
High CPU `%sys`	`perf top` for kernel symbols	`bpftrace` on hot kernel path
CPU `%steal` > 0	`turbostat`, cloud provider metrics	Escalate; no host-side fix
OOM kills in dmesg	`free -m`, `/proc/pressure/memory`	`smem -t`, `valgrind` (dev)
Slow disk (elevated await)	`iostat -xz 1` (await, aqu-sz)	`iotop`, `blktrace`, `fio` baseline
Slow random I/O	`iostat` IOPS vs `fio` baseline	`biolatency-bpfcc` histogram
High network latency / packet loss	`ss -s` (retransmits), `ping`	`ethtool -S`, `tc qdisc`, `nstat`
NIC saturation	`sar -n DEV %ifutil`, `ethtool -S tx_dropped`	Increase txqueuelen, NIC ring buffer
Slow application (unknown cause)	`perf top` (CPU hotspots), `strace -c` (syscall time)	Flamegraph, `bpftrace` one-liners
Memory leak suspicion	`/proc/<pid>/status VmRSS` over time, `AnonPages` trend	`smem -t`, `valgrind` (dev env)
Mysterious file access	`opensnoop-bpfcc`, `strace -e openat`	`lsof +D /path`
New unexpected connections	`tcpconnect-bpfcc`	`ss -tp`, `strace -e network`
Scheduler latency (tail latency)	`runqlat-bpfcc`	Tune CPU affinity, cgroup cpu.shares

Installation Reference — Ubuntu 24.04 / Debian 12

# perf
sudo apt install linux-tools-$(uname -r) linux-tools-generic

# bpftrace (version 0.24.x in current repos)
sudo apt install bpftrace

# BCC tools (opensnoop, biolatency, runqlat, etc.)
sudo apt install bpfcc-tools linux-headers-$(uname -r)

# Tools are installed as: /usr/sbin/<tool>-bpfcc
# Example: /usr/sbin/runqlat-bpfcc, /usr/sbin/opensnoop-bpfcc

# sysstat (vmstat, iostat, sar, pidstat, mpstat)
sudo apt install sysstat

# smem
sudo apt install smem

# iotop
sudo apt install iotop

# htop, fio
sudo apt install htop fio

Putting It Together

A methodical investigation of a slow system looks like this:

Run the first-60-seconds checklist. By the end, you have numbers for CPU utilization, run queue depth, memory availability, swap activity, disk await and queue depth, and network throughput and retransmit rates. You have a hypothesis.
Apply the USE method to the resource your checklist implicated. If disk await is elevated: is utilization the problem (sustained 100% %util) or is saturation the problem (aqu-sz > 1 but %util looks moderate)? The distinction matters for the fix.
Drill down with the appropriate specialized tool. CPU hypothesis: perf record -g and a flamegraph. Memory hypothesis: /proc/meminfo, PSI in /proc/pressure/memory, smem. Disk hypothesis: iotop to find the process, then blktrace or biolatency-bpfcc to understand the latency distribution. Network hypothesis: ethtool -S for hardware errors, nstat for protocol counters.
For anything that does not fit a standard hypothesis: reach for bpftrace. The ability to write a probe in 30 seconds that answers a precise question — “how many bytes is this process reading per read() call?” or “which functions is the kernel calling when this process sleeps?” — is what separates modern Linux performance engineering from the traditional approach of running fixed tools and hoping for insight.
Correlate across layers. A symptom visible at the application level (slow query) may have a cause at a completely different layer (CPU throttling due to thermal limits, kernel lock contention in the memory allocator, NIC driver dropping packets). The tool map exists precisely because you need to systematically traverse the layers rather than assume the cause lives where the symptom appears.

Performance analysis is not a collection of commands. It is a process of forming hypotheses and eliminating them efficiently. The tools are the instruments; the method is what makes you useful under pressure.