perf: Linux's Best Profiler

Every serious Linux performance investigation eventually ends at perf. Not because it’s pretty — the man page is vast, the UI is terse, and the output takes real study to read — but because perf is the tool that actually shows you what the CPU is doing. Flamegraphs, eBPF tools, Java agents, Python profilers: many of them either wrap perf directly or exist to expose data that perf collects.

This post is the practical guide I wish I had when I first started profiling Linux systems seriously. It covers the subcommands you’ll actually use (stat, top, record/report, script), what performance counters mean and how to read them, the knobs that catch everyone (frame pointers, kernel permissions, symbol resolution), and enough of the CPU microarchitecture vocabulary to make sense of what perf tells you.

What perf actually is

perf is a Linux kernel performance analysis framework with a userspace CLI. The kernel exposes performance monitoring units (PMUs) — hardware counters on the CPU that track events like cycles, cache misses, branch mispredictions, and much more — plus software events (context switches, page faults, scheduler events) and tracepoints (static instrumentation points throughout the kernel).

Two pieces make perf unusually powerful:

1. Sampling: the PMU is programmed to interrupt the CPU every N events (say, every 10 million cycles). At each interrupt, the kernel grabs a stack trace. Over a few seconds of sampling, the distribution of stack traces tells you where CPU time is going — with hardware-level accuracy and extremely low overhead.

2. Counting: instead of interrupting, the counter simply accumulates. At the end you get “this process executed 12.3 billion cycles and 8.1 billion instructions.” This is how perf stat tells you the IPC (instructions per cycle) of your workload.

Both modes work on kernel code and userspace code simultaneously. Unlike language-specific profilers that only see one layer, perf shows you the whole stack: your function called memcpy which called __copy_to_user which ran 2.4 million cycles, and here’s the split between user and kernel time.

Installing and permissions

On Debian/Ubuntu: apt install linux-tools-$(uname -r) (or linux-tools-generic). On RHEL/Fedora: dnf install perf. On most distros, the binary is versioned to your kernel — old perf against new kernel may be missing events.

The permissions model is where everyone stumbles first. By default, sampling kernel addresses requires CAP_PERFMON or root. Two sysctls control this:

1
2
3
4

# Allow users to sample their own processes (including kernel frames)
sudo sysctl -w kernel.perf_event_paranoid=-1
# Allow resolving kernel symbols
sudo sysctl -w kernel.kptr_restrict=0

perf_event_paranoid levels:

• 3 (default on many distros) — no access to perf_event_open without CAP_PERFMON.
• 2 — users can monitor their own processes, no kernel samples.
• 1 — users can monitor their own processes with kernel samples.
• 0 — users can monitor system-wide events.
• -1 — unrestricted.

On a shared production host you probably can’t drop paranoid to -1. On a dev box or a dedicated performance-testing host you should. The difference between having kernel stacks and not is enormous — you’ll mistake “my app is slow” for “syscalls are slow” without them.

perf stat: the first thing to run

When you want to know “is this workload CPU-bound, memory-bound, or just slow?” — perf stat is the answer.

$ perf stat ./my_program

 Performance counter stats for './my_program':

          1,452.31 msec task-clock                #    0.998 CPUs utilized
                 9      context-switches          #    6.197 /sec
                 0      cpu-migrations            #    0.000 /sec
             2,103      page-faults               #    1.449 K/sec
     5,812,456,123      cycles                    #    4.002 GHz
     9,431,872,904      instructions              #    1.62  insn per cycle
     1,823,441,221      branches                  #    1.256 G/sec
        43,211,678      branch-misses             #    2.37% of all branches

       1.454761239 seconds time elapsed
       1.441000000 seconds user
       0.009000000 seconds sys

A few things to read from this:

• IPC (instructions per cycle): 1.62 here. Modern CPUs can retire up to 4–6 instructions per cycle. Below 1.0 means the CPU is stalling on something — cache misses, branch mispredictions, or memory latency. Above 3.0 means your code is well-vectorized and cache-friendly.
• Branch miss rate: 2.37%. A well-predicted branch-heavy workload runs below 1%. Above 5% suggests data-dependent branches (unpredictable input) or pointer chasing.
• Frequency: 4.002 GHz. If this drops well below the nominal boost clock, you’re thermally throttled or frequency-scaled down. Check cpupower frequency-info.

Add -d for “detailed” mode which includes cache stats:

$ perf stat -d ./my_program
...
     2,341,892,773      L1-dcache-loads
        94,112,438      L1-dcache-load-misses     #    4.02% of all L1-dcache accesses
        26,743,219      LLC-loads
         8,193,421      LLC-load-misses           #   30.65% of all LL-cache accesses

LLC (last-level cache) miss rate over 10% is a flag — you’re hitting main memory a lot. 30% is dire. Candidates: the working set is bigger than cache, cache-unfriendly access patterns (linked lists, random pointer chasing), or false sharing between threads.

-e lets you pick specific events:

1
2
3

perf stat -e cycles,instructions,cache-misses,cache-references,dTLB-load-misses \
    -e stalled-cycles-frontend,stalled-cycles-backend \
    ./my_program

perf list shows the full catalog of supported events. It’s long; on an Intel CPU, expect thousands. The important categories:

• Hardware events: cycles, instructions, cache-misses, branch-misses, bus-cycles.
• Software events: context-switches, cpu-migrations, page-faults, minor-faults, major-faults.
• Cache events: L1-dcache-loads, L1-icache-load-misses, LLC-loads, dTLB-load-misses, etc.
• Raw PMU events: vendor-specific, accessible via r prefix plus event code. Rarely needed directly — use the named events.

perf top: htop for function-level CPU

perf top is an interactive view of where CPU time is going in real-time across the whole system.

$ sudo perf top
   12.34%  [kernel]        [k] _raw_spin_unlock_irqrestore
    8.12%  my_program      [.] hash_combine
    6.41%  libc-2.31.so    [.] __memcpy_avx_unaligned_erms
    5.77%  [kernel]        [k] copy_user_enhanced_fast_string
    ...

Each row is a function and the percentage of samples that landed inside it. [.] means userspace, [k] means kernel, [g] means guest (VM).

Press a on any row to get an annotated view — perf disassembles the function and shows per-instruction sample counts. This is how you learn that 60% of your “slow function” is actually spent on two specific load instructions that are missing the L1 cache.

perf top -p <pid> narrows to a single process, which is what you want 90% of the time.

perf record and perf report: the workhorse

perf top is live. perf record captures a session to disk for later analysis.

1
2
3
4
5
6
7
8

# Sample the whole system for 10 seconds at 99Hz with call stacks
sudo perf record -F 99 -a -g -- sleep 10

# Sample a specific process
perf record -F 99 -p <pid> -g -- sleep 30

# Sample a specific command
perf record -F 99 -g ./my_program arg1 arg2

Key flags:

• -F 99 — sample frequency in Hz. 99 is a classic choice (prime to avoid aliasing with 100-Hz timers). Higher frequencies catch more detail at more overhead. 999 Hz is about the practical ceiling.
• -a — system-wide. All CPUs, all processes.
• -p <pid> — single process.
• -g — record call stacks. Without this you get flat per-function counts; with it you get a tree of who called whom.
• --call-graph=dwarf or --call-graph=fp or --call-graph=lbr — how to unwind stacks. This is the critical decision.

Stack unwinding methods:

1. Frame pointers (fp): the classic approach. Requires code compiled with -fno-omit-frame-pointer. Fast, cheap, reliable — but modern distros built their libraries without frame pointers for years, which means you can sample your code and see no stack above it. The Fedora / Ubuntu push in 2023-24 to re-enable frame pointers system-wide was specifically to fix this. If you’re on a modern distro, frame pointers are likely available; on older ones, they’re not.

2. DWARF (dwarf): unwinds using debug info. Works without frame pointers but requires libunwind, eats more CPU during recording, and produces larger perf.data files. Pass --call-graph=dwarf,65528 to set a larger stack capture size (the default truncates at 8KB, which isn’t enough for deeply-nested Python or Java).

3. LBR (lbr): Last Branch Records. The CPU itself records recent branches in a ring buffer. Very fast, low overhead, but limited to 16–32 branches of history on most CPUs — only shows the most recent few frames. Good for hot, shallow call stacks.

My defaults: frame pointers when available, DWARF when not. LBR for very-high-frequency sampling where DWARF overhead matters.

After recording: perf report.

$ perf report --stdio
# Samples: 12K of event 'cycles:P'
# Event count (approx.): 45823942198
#
# Children      Self  Command  Shared Object      Symbol
# ........  ........  .......  .................  ..........................
    34.12%    22.87%  worker   my_program         [.] compute_hash
    34.12%    22.87%         |
                              --22.87%--compute_hash
                                 |
                                 |--8.41%--__memcpy_avx_unaligned_erms
                                 |
                                 --3.84%--_int_malloc

• Self: samples in this function, excluding children.
• Children: samples in this function and its descendants.
• The tree shows “this function was sampled here, called by that function X% of the time.”

Interactive mode (no --stdio) is usually better. Arrow keys to navigate, Enter to expand, a to annotate a function, / to search.

Flamegraphs: the pattern everyone converged on

Textual perf report is dense. Brendan Gregg’s flamegraph trick flips it: each stack trace becomes a vertical stack of rectangles, the width of each rectangle is proportional to how often that stack appeared, and the result is an image that reveals hot paths at a glance.

1
2
3

perf record -F 99 -g -p <pid> -- sleep 30
perf script > out.stacks
stackcollapse-perf.pl out.stacks | flamegraph.pl > flame.svg

The generated SVG is interactive — click to zoom, search for function names, hover for details. Flamegraphs are the single best way to communicate a CPU profile to someone who isn’t sitting with you reading perf report output. Ship them as part of perf investigations.

For a sharper tool, check out the Rust-based inferno reimplementation (inferno-collapse-perf, inferno-flamegraph) — drop-in, much faster on large captures.

Symbol resolution: the most common source of confusion

perf report showing [unknown] instead of function names is the #1 question on Stack Overflow about perf. The flowchart:

1. The binary has no symbols. Stripped release binary. strip --strip-debug --keep-symbol=main is the modern production move that preserves symbols for profiling; if the binary was fully stripped, you need a separate debuginfo package or a debug build.

2. Kernel symbols are hidden. kptr_restrict=1 (default) prevents non-root from seeing kernel symbols even with perf samples. Set kptr_restrict=0 or run as root.

3. Symbols are in a separate debuginfo package. On RHEL/Fedora, dnf debuginfo-install glibc. On Ubuntu, apt install libc6-dbg. Perf looks up symbols via the build-ID matching mechanism.

4. JIT-compiled code (Java, Node, Python with Numba, V8) — the runtime creates anonymous machine code that has no ELF symbols. You need a jitdump file (perf-$PID.map or a proper jit-$PID.dump) generated by the runtime. Java: -XX:+PreserveFramePointer -agentpath:libperf-map-agent.so. Node: --perf-basic-prof. Without this, you see 0x7f... hex addresses and nothing useful.

5. Container weirdness — perf running on the host can’t find symbols inside a container’s filesystem unless you bind-mount the container root or run perf inside the container. perf buildid-cache can help.

Reading a CPU-bound profile

Here’s what I look for, roughly in order:

1. Is IPC > 1.5 on a modern CPU? If yes, the CPU is doing real work; time spent is time the algorithm needs. Optimize at the algorithm level or with vectorization.

2. Is IPC < 1.0? Something is stalling. Look at perf stat -d for cache miss rates and branch-miss rates.

3. High LLC miss rate? Working set too big, bad data layout, or false sharing. Tools: perf c2c detects false sharing specifically.

4. High branch miss rate? Data-dependent branches. Options: reduce branching (branchless code, SIMD), make branches predictable (sort data before branching on it), or eliminate the branch (table lookup).

5. High kernel time in sys column? Lots of syscalls, scheduling, or I/O. perf trace or strace narrows it down.

6. Large stalled-cycles-frontend? The CPU can’t fetch or decode instructions fast enough. Often means code bloat — too much code pressure on the I-cache. Consider link-time optimization, PGO, or just less code.

7. Large stalled-cycles-backend? Execution is waiting on memory or ALUs. Usually memory, usually means cache misses. Back to point 3.

Top-down microarchitecture analysis

On Intel CPUs (Nehalem+) and AMD (Zen+), the PMU supports Top-down Microarchitecture Analysis (TMA), a structured way to bucket cycles into one of four categories: Retiring, Bad Speculation, Frontend Bound, Backend Bound.

$ perf stat -M TopDownL1 ./my_program
...
  Retiring              35.2%   (executing useful work)
  Bad Speculation        4.1%   (mispredicted branches)
  Frontend Bound        12.3%   (fetching/decoding stalls)
  Backend Bound         48.4%   (memory or ALU stalls)

Backend Bound at 48% means the CPU is waiting on memory or execution resources roughly half the time. The next level (TopDownL2) breaks this into Memory Bound vs Core Bound, and subsequent levels get even more specific. This is the single most structured way to diagnose “what is the CPU actually doing” for a given workload.

perf script: custom processing

perf script dumps raw samples as text — useful when you want to pipe through your own tools.

1
2
3
4
5
6
7

$ perf script
my_program 1234 [003] 192837.482913: 10000000 cycles:ppp:
        ffffffff810af3c2 __entry_text_start+0x2
        55c4e213b4b5 compute_hash+0x15 (/home/user/my_program)
        55c4e213b691 process_batch+0xa1 (/home/user/my_program)
        55c4e213b7a8 worker_main+0x58 (/home/user/my_program)
        55c4e213b924 main+0x24 (/home/user/my_program)

Each record is a sample: pid, tid, CPU, timestamp, weight, event, and stack. Pipe to flamegraph, to a CSV converter, to whatever. This is where perf becomes composable with your own tools.

perf script -F comm,pid,time,ip,sym,dso lets you customize the output columns.

Sampling other things: not just cycles

perf isn’t only about CPU cycles. You can sample on any event. Useful ones:

perf record -e cache-misses — “where are my cache misses coming from?” Samples land in code that was accessing memory that missed. Much more actionable than global cache miss counters.

perf record -e major-faults — page faults that hit disk. Samples show you which code paths are triggering disk I/O via memory-mapped files.

perf record -e sched:sched_switch — context switches. Tells you which processes are frequently preempting each other, useful for diagnosing noisy neighbors.

perf record -e block:block_rq_issue — block I/O requests. Shows exactly which code is issuing disk I/O.

perf record -e syscalls:sys_enter_openat — every openat syscall system-wide. Shows you which files your code is thrashing open.

Event-based profiling is where perf becomes almost magical. You’re not limited to “where is CPU time spent” — you can ask “where are my cache misses, from which code path, triggered by which data.”

perf mem and perf c2c: memory access analysis

For workloads suspected of having memory-layout problems, two specialized commands:

perf mem samples memory accesses with their latency. Post-process to find which instructions are causing the longest stalls on memory. This directly identifies cache misses causing performance problems — and distinguishes L1, L2, L3, and DRAM hits in the latency histogram.

perf c2c (cache-to-cache) detects false sharing: two CPUs writing to different variables that happen to live in the same cache line. Each write invalidates the other CPU’s copy. Performance collapses for no obvious reason — the algorithm is embarrassingly parallel but throughput degrades with more threads. perf c2c shows you which cache lines are ping-ponging between cores.

Both require Intel-specific PMU features (PEBS) that aren’t available on AMD, though AMD’s IBS provides similar data via different commands.

perf trace: strace without the overhead

strace ptrace-attaches and slows the traced process dramatically. perf trace uses tracepoints and has overhead closer to 1–5%, usable in production:

1
2
3

perf trace -p <pid>
perf trace -e 'syscalls:sys_enter_open*' -a
perf trace --summary -p <pid>   # aggregated syscall stats

For a quick “what syscalls is this process making” on a production-sensitive workload, perf trace is the correct tool.

Overhead: how much does profiling cost?

Sampling perf is one of the lowest-overhead profilers available:

• At 99 Hz with frame-pointer unwinding, overhead is well under 1%.
• At 999 Hz with DWARF, closer to 3–5%.
• Event-driven profiling (cache misses, branch misses at high frequency) can hit 10%+.
• perf trace on a noisy syscall tracepoint can be significant; narrow the filter.

This is generally safe to run in production, unlike strace or heavyweight agents. I routinely run perf record -F 99 -g on busy production hosts for 30 seconds at a time to diagnose live issues.

Practical workflow for a performance investigation

A typical session:

1. Baseline with perf stat. Get IPC, cache miss rate, branch miss rate. Know what regime you’re in before sampling.
2. perf top on the target process. What functions are hot?
3. perf record -F 99 -g -p <pid> -- sleep 30 for a full profile.
4. Flamegraph the result. Look at the overall shape.
5. perf report interactively. Drill into the hot functions. Annotate (a) to see per-instruction counts.
6. If IPC was low: perf stat -d -d -d for detailed cache/branch/frontend/backend breakdown. TopDownL1 if Intel/AMD Zen.
7. If LLC misses were high: perf mem to find the offending instructions. perf c2c if threaded and showing unexplained contention.
8. If kernel time was high: perf trace --summary to see syscall distribution, then targeted tracepoints for the hot ones.
9. Change something. Remeasure. Commit the fix with the before/after numbers in the commit message.

Most production performance problems get solved in that first flamegraph plus perf stat. Going deeper into PMU details is for the cases where the flamegraph says “this function is hot” but the function looks innocuous — which means the hot instructions are doing something expensive you can’t see from source (cache miss, branch miss, microcode trap).

Gotchas and things that waste hours

• perf data is huge. A 30-second system-wide record at 999 Hz with DWARF can be tens of GB. Narrow scope. Use frame pointers. Sample less frequently.
• Kernel samples attributed to wrong functions on syscall entry/exit. Known artifact of how sampling interrupts interact with syscall boundaries. Trust the broad picture, distrust single-sample decisions.
• Virtualization hides PMU events. Many hypervisors don’t expose hardware counters to guests. perf stat shows <not supported> for most events. Workaround: use --host perf on the hypervisor, profile guest code from there, or enable vPMU passthrough.
• Clock frequency scaling confuses IPC math. If the CPU drops from 4.0 GHz to 2.0 GHz for thermal reasons during your recording, IPC averages become meaningless. Pin the frequency (cpupower frequency-set -f 4000000) for benchmarking.
• NUMA mismatches. A process running on NUMA node 0 accessing memory allocated on NUMA node 1 pays huge latency penalties that show as backend stalls. numactl --hardware tells you the topology, numactl --cpubind=0 --membind=0 ./prog pins appropriately.
• Container permissions. Running perf inside a container requires --cap-add=SYS_PERFMON (recent) or --cap-add=SYS_ADMIN (older) plus access to /proc/kallsyms and /sys/kernel/debug. Profiling from the host is usually simpler.

Where perf fits in 2026

perf isn’t the newest profiler. eBPF-based tools (bpftrace, bcc-tools) can answer many of the same questions with more flexibility. async-profiler for JVM, py-spy for Python, cargo-flamegraph for Rust — language-specific profilers often have better UX. In practice, perf remains the base layer:

• eBPF tools are more flexible but higher-overhead for broad sampling. Perf’s PMU sampling is unmatched for general “where does CPU time go.”
• Language profilers excel at their language but lose the kernel view. Perf connects userspace to kernel in one profile.
• Commercial tools (Intel VTune, AMD uProf, Arm Streamline) have richer analysis UI. Many use perf under the hood anyway.

For most Linux performance investigations, perf is still the first tool to pick up. The investment in learning it compounds — the mental model (PMU events, sampling vs counting, kernel/user boundaries) transfers to every other profiling tool you’ll encounter. A week of real engagement with perf changes how you think about performance for the rest of your career.