Flamegraphs: Reading and Generating

Reading a stack profile as text is like reading a waterfall diagram one line at a time. You can do it, but the shape of the problem — which paths dominate, which call stacks are identical, where the CPU is really spending time — doesn’t pop until you see it rendered.

Brendan Gregg’s flamegraph visualization was the breakthrough that turned profiling from a specialist activity into something engineers actually enjoy. The trick is deceptively simple: take every stack trace from your profile, stack them up vertically, merge identical prefixes, and color them. The wide boxes at the top are where time is being spent. Anyone on your team can point at a 40% wide yellow bar and say “that’s the hot path.”

This post is the practical guide to flamegraphs: how to read them correctly (including the common misreadings), how to generate them from every major profiler, what the different variants (differential, off-CPU, icicle) tell you, and how to avoid the pitfalls that make many flamegraphs lie.

What a flamegraph actually is

The simplest definition: a flamegraph is a visualization of stack traces sampled from a profile, with identical stack prefixes merged horizontally.

Mechanically:

1. Start with raw samples. Each sample is a stack trace — a list of function frames from outermost (e.g., main) to innermost (the leaf function running when the sample was taken).
2. Group stacks by their identity. If 100 samples have the exact same stack, merge them into one entry with weight 100.
3. Render as nested rectangles. The leaf function is at the top; the caller is below; its caller is below that; all the way down to the root.
4. Width is proportional to number of samples. A function 40% as wide as the whole graph was sampled in 40% of samples.
5. Color has no meaning by default. It’s usually randomized for visual differentiation.

That’s it. No axes, no time dimension (samples are aggregated), no causality between neighboring bars. Just “these are the stacks we saw, this is how often we saw them.”

Reading a flamegraph correctly

The most common mistake is reading left-to-right as if it were a timeline. It isn’t. The x-axis is alphabetical (by function name) within each level. A stack that appears on the left of another doesn’t mean it happened first; samples are simply sorted for readability.

The correct reading:

1. Width = time. Wider means more samples means more CPU time (or whatever event you sampled on).
2. Height = stack depth. Taller doesn’t mean slower. A deep stack is just a deeply-nested call chain.
3. The top edge is where the CPU is. The leaf function is what was running at each sample. If function foo has a wide flat top, foo itself is hot. If foo has most of its width attributed to a child at the top of its stack, foo is only hot because it calls expensive things.

The question “where is CPU time going?” is answered by looking at the top edge of the graph — the skyline. Walk the skyline left to right; wherever it’s wide and flat, that’s where the CPU is spending cycles.

Three shapes worth recognizing:

• Flat plateau at the top. A function with no expensive children but lots of CPU time in its own body. Candidates: tight loops, hot math, memory operations.
• Tall narrow spike. A deeply-nested call stack that’s not actually that hot. Looks impressive, usually not the problem.
• Pyramid. A function that calls many different things, each taking some fraction of time. The function’s overall width is the total work; the width of each child shows what it’s spending time on.

Patterns to look for:

• Unexpected wide bar deep in the call tree. Something is being called way more than it should. Often an algorithmic problem — an O(n²) loop, a cache that isn’t caching, a retry storm.
• Wide bar labeled [unknown] or a hex address. Symbol resolution failed. Sometimes this is a JIT frame, sometimes a stripped binary. Fix the symbol problem before drawing conclusions.
• Wide bar in system libraries (libc, libssl). Not inherently bad — memcpy doing lots of work means you’re copying lots of data, which might be normal. Check the caller in the layer below to understand why.
• Lots of narrow towers. High call stack churn — tons of different paths, none dominating. Either the workload is genuinely diverse, or your sampling rate is too low to find the hot path.

Generating flamegraphs from perf

The canonical toolchain for Linux CPU profiling:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# 1. Sample
perf record -F 99 -g -p <PID> -- sleep 30

# 2. Dump stacks as text
perf script > out.stacks

# 3. Collapse identical stacks
stackcollapse-perf.pl out.stacks > out.folded

# 4. Render
flamegraph.pl out.folded > flame.svg

# 5. Open in a browser
firefox flame.svg

The scripts come from Brendan Gregg’s FlameGraph repository. They’re pure Perl, zero dependencies, and do exactly one thing each. The folded format between steps 3 and 4 is a simple one-line-per-unique-stack format:

main;process_batch;compute_hash;hash_combine 1523
main;process_batch;compute_hash;__memcpy_avx_unaligned_erms 892
main;process_batch;read_input;__read 234

Semicolon-separated frames, space, sample count. This format is what every other profiler eventually exports to — it’s the lingua franca of stack aggregation.

Inferno: the Rust rewrite that you probably want

The original Perl scripts work but are single-threaded and can be slow on large captures (gigabytes of stacks). Inferno is a drop-in Rust reimplementation that’s roughly 10–40× faster:

1
2
3

cargo install inferno

perf script | inferno-collapse-perf | inferno-flamegraph > flame.svg

Same output format, much better performance. For anyone doing regular flamegraph work, switch.

Inferno also integrates with other profiler formats via inferno-collapse-* helpers: DTrace stacks, VTune, Visual Studio, sample, etc. And the inferno-diff-folded subcommand generates differential flamegraphs (more on those below).

Differential flamegraphs: “what changed?”

A single flamegraph tells you where CPU is going right now. A differential flamegraph compares two profiles and shows you what changed between them. This is how you answer “is the new version faster?” or “what started eating CPU after the deploy?”

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Capture before
perf record -F 99 -g -p <PID> -- sleep 30
perf script | stackcollapse-perf.pl > before.folded

# Deploy new version

# Capture after
perf record -F 99 -g -p <PID> -- sleep 30
perf script | stackcollapse-perf.pl > after.folded

# Diff
difffolded.pl -n before.folded after.folded | flamegraph.pl > diff.svg

Colors change meaning in a differential flamegraph: red means the stack got hotter in after.folded, blue means it got cooler. Width is typically the max between the two profiles.

Differential flamegraphs are extraordinarily effective for release-to-release comparisons. A 2%-wide red patch inside an otherwise unchanged graph tells you exactly where your regression lives.

Off-CPU flamegraphs: what your process is waiting for

Standard CPU flamegraphs only show time when the CPU is actively executing. They don’t show time spent blocked — waiting for I/O, sleeping on a lock, stalled on a syscall. For many real workloads, that blocked time is where the latency lives.

Off-CPU flamegraphs invert the question. Instead of “what is the CPU doing?” they ask “what is this process waiting for, and for how long?”

Mechanics: hook scheduler tracepoints (specifically sched:sched_switch) to capture a stack trace every time the scheduler takes a thread off-CPU, plus the stack from when it’s placed back on-CPU. The difference in timestamps is the off-CPU time. Aggregate by stack; flamegraph the result.

With bpftrace:

1
2

offcputime-bpfcc -p <PID> -df 30 > out.stacks
flamegraph.pl --color=io --title="Off-CPU" out.stacks > offcpu.svg

Or using the BCC tools (offcputime, offwaketime). The typical signatures in an off-CPU flamegraph:

• Wide futex_wait — waiting on a mutex or condition variable. Threads are blocking on synchronization.
• Wide io_schedule — waiting on disk I/O. Probably a read from slow storage.
• Wide schedule_timeout — sleep calls. Intentional waits, usually from periodic tasks.
• Wide poll_schedule_timeout — waiting on a file descriptor (socket, pipe). Network I/O or IPC.

Off-CPU flamegraphs routinely expose contention invisible in CPU flamegraphs. A server that’s “only 20% CPU utilized” but handles fewer requests than expected often has threads spending most of their time in futex_wait — a lock serializing work.

The one caveat: off-CPU tracing overhead scales with context switch rate. On a system doing millions of context switches per second, the tracing itself can measurably slow the workload. Keep capture windows short.

Icicle graphs: flipped flamegraphs for top-down reading

An icicle graph is a flamegraph drawn upside down: the root is at the top, leaves at the bottom. Width still represents time, but now the skyline is at the bottom and the hierarchy reads naturally top-down.

Icicle graphs are easier for some people to read (the outer function is on top, matching how we usually write code). They’re identical information, just a different orientation. Many modern profiler UIs (pprof, speedscope, Grafana Pyroscope) default to icicle.

flamegraph.pl --inverted produces icicle graphs from the same folded data.

Profiling non-C languages

Flamegraphs shine when your profile has real symbol names. That’s where language-specific tricks matter.

Java / JVM

Default JVM profiles stop at the JNI boundary — below that, you see assembly addresses with no symbols. Fixes:

• async-profiler: the industry standard for JVM profiling. Wraps perf_events and JVMTI to produce unified stack traces.

1

java -jar async-profiler.jar -d 30 -f flame.html <PID>

Generates an HTML flamegraph directly. Supports CPU, allocation, lock, and wall-clock profiling modes. Reliably resolves both Java and native frames.

• Old-school approach: -XX:+PreserveFramePointer plus perf-map-agent to emit a perf-<PID>.map file mapping JIT addresses to Java method names. Standard perf record + stackcollapse-perf.pl then works.

Python

CPython profiles are dominated by the interpreter loop unless you use a tool that understands Python frames.

• py-spy: samples CPython’s frame objects via ptrace. Produces flamegraphs directly.

1

py-spy record -o flame.svg --pid <PID> --duration 30

Very low overhead, no code changes required. Handles subprocess, threads, and async reasonably.

• Austin: another sampling profiler, integrates with perf’s folded format directly.

Node.js / V8

• --perf-basic-prof: Node emits a JIT symbol map file that perf reads. Combine with standard perf record.
• 0x: a wrapper that does all of this for you plus renders an HTML flamegraph.

Go

• pprof: Go’s built-in profiler generates flamegraph-friendly data. go tool pprof -http=:8080 cpu.prof opens a browser view with multiple visualizations including flamegraph.
• runtime/trace: for scheduling, goroutine, and I/O timing — complements pprof.

Rust

• cargo flamegraph: wraps perf for Rust binaries with symbol handling done automatically. Often sufficient.

Grafana Pyroscope and continuous profiling

Flamegraphs used to be something you generated manually during an incident. Continuous profiling changes that: a background agent samples constantly, stores historical profiles, and lets you view a flamegraph for any time range.

Grafana Pyroscope (previously standalone, now part of the Grafana stack) is the leading open-source continuous profiler. Agents exist for Go, Java, Python, Node, Rust, Ruby, and .NET. Profiles are tagged with service/environment/pod labels, searchable the same way you search logs.

The value isn’t just the always-on aspect. It’s that you can:

• Compare two time ranges (differential flamegraph).
• Correlate profiles with metrics and traces (Grafana’s unified UI ties them together).
• Find the exact moment a regression started.

If your org has a Grafana stack, Pyroscope is the natural next step. For AWS-heavy shops, CodeGuru Profiler serves a similar role. Commercial options (Datadog Continuous Profiler, New Relic CodeStream, Pyroscope Cloud) are mature.

speedscope: the interactive viewer

flamegraph.pl outputs a static SVG. For deeper exploration, speedscope is a web-based interactive viewer that supports multiple formats (pprof, Chrome trace, folded stacks, async-profiler, and more).

Drop a profile file onto speedscope.app — or run it locally — and you get:

• Time-ordered view (not aggregated).
• Aggregated flamegraph.
• Left-heavy flamegraph (similar stacks grouped for pattern-matching).
• Sandwich view (select a function, see its callers above and callees below).

The sandwich view is the killer feature. If you’ve ever wanted to ask “who is calling memcpy and how much time is each caller spending?” — that’s exactly what sandwich view shows you.

Common lies that flamegraphs tell

Flamegraphs are visualizations of sampled data. The quality of the picture depends on the quality of the samples. Common failure modes:

Missing frame pointers. On distros that built without -fno-omit-frame-pointer (most pre-2023 Ubuntu/Debian/Fedora), perf’s frame-pointer unwinding sees only one or two frames of userspace and then the stack vanishes. Result: big “flat” frames at low levels representing “we don’t know what this code is doing.” Fix by using DWARF unwinding (--call-graph=dwarf), installing frame-pointer-enabled builds of the distro (Fedora 38+, Ubuntu 24.04+), or compiling your own binaries with frame pointers enabled.

Inlined functions. Compilers inline small functions aggressively. Inlined functions don’t appear in stack traces at all — you see the parent function doing the work of the inlined child. Cost: blame is attributed to the caller. Fix: compile with -fno-inline for profiling (obviously changes performance characteristics, so use judiciously). Alternatively, check the debug info — DWARF-based unwinders can reconstruct inline frames.

Sample rate aliasing. At 100 Hz, samples happen every 10 ms. A function that runs in 3 ms, 20,000 times per second, is never sampled in its body — samples only catch what’s running at the 10 ms ticks. Increase sample rate (999 Hz) or use event-based sampling for short hot functions.

Wall-clock vs CPU time. Standard CPU flamegraphs only count time when the CPU is executing. A program that spends 90% of wall-clock waiting for I/O and 10% computing will show the 10% computing, amplified to 100% of its flamegraph. This can make a slow-because-it-waits program look like a slow-because-it-computes program. Use off-CPU flamegraphs or wall-clock profilers (async-profiler’s wall mode) when latency is the concern.

Symbolication gaps. Stripped binaries, missing debuginfo, container symbol mismatch. Always inspect the SVG and check that function names look sensible before drawing conclusions.

Other flamegraph variants

Allocation flamegraphs. Color by allocation size / count instead of CPU time. Shows where memory is being allocated. Async-profiler’s -e alloc mode produces these directly.

Lock contention flamegraphs. Sample on lock acquisition events. Shows which code paths are contending on which locks. Async-profiler’s -e lock mode.

Memory leak flamegraphs. Sample malloc/free calls and render only the allocations that weren’t freed by profile end. Identifies leak origins.

GPU flamegraphs. Emerging area — NVIDIA’s Nsight Systems can export flamegraph-compatible profiles for CUDA kernels. Still less mature than CPU flamegraphs.

IO flamegraphs. Sample block layer tracepoints. Shows what code is driving disk I/O. iosnoop + flamegraph scripts.

The pattern generalizes: any stack-tagged event stream can be flamegraphed. CPU time is the most common, but anything where the question is “show me the distribution of these events by call stack” benefits.

Generating flamegraphs in CI

A mature performance practice is generating flamegraphs for every release. Integrate into CI:

1. Run a representative workload against the build.
2. Collect a profile.
3. Render both a flamegraph and a differential against the previous release.
4. Upload as a build artifact, or post to a Grafana dashboard.
5. Optionally: automated regression detection (large red areas > threshold → alert).

For services running in staging, an always-on continuous profiler (Pyroscope) plus alerting on significant flamegraph shape changes catches performance regressions before they reach production.

Reading flamegraphs in team settings

A social observation: flamegraphs are unreasonably effective at getting non-specialists to engage with performance work. Showing a wide yellow bar and saying “this function is 38% of our CPU” converts in minutes what used to take a 30-minute text-profile walkthrough.

Practices that make this effective:

• Annotate the SVG. Add a title, the service name, the time range, the sample count. A flamegraph with no context is useless in three months.
• Save the folded data, not just the SVG. The SVG can’t be re-analyzed, re-rendered, or diffed. The folded file can.
• Commit flamegraphs to the repo. For major performance work, including the baseline and post-optimization flamegraphs in the PR makes the benefit concrete.
• Use the interactive SVG. The default flamegraph.pl SVG has built-in zoom and search. Clicking a frame zooms to just its subtree; Ctrl+F searches for function names. Reviewers who don’t know this miss half the value.

A practical workflow

For a typical performance investigation:

1. CPU flamegraph first. perf record -F 99 -g -p <PID> -- sleep 30 + inferno. Get the shape of the problem.
2. Look at the skyline. Identify the widest leaf functions. Those are your hot spots.
3. Navigate the SVG. Click the hot function to zoom; search for suspected culprits.
4. If the CPU graph looks fine but the app is slow: off-CPU flamegraph. Something is waiting.
5. If an allocation is suspected: allocation flamegraph (async-profiler for Java, pprof allocation profile for Go).
6. Make a change. Re-profile. Differential flamegraph. Confirm the fix actually helped.
7. Attach both SVGs to the PR. Reviewers can see the improvement visually.

Where flamegraphs fit — and don’t

Flamegraphs are excellent for:

• CPU-bound work where the hot path matters.
• Understanding unfamiliar code (“what does this service actually spend time on?”).
• Before/after comparisons.
• Communicating performance issues to teammates.

They’re weak or misleading for:

• Latency-sensitive workloads where tail latency (not mean) is what matters. Flamegraphs aggregate; tail latency lives in the rare case.
• I/O-bound workloads unless you use off-CPU or wall-clock profiling.
• Problems that only appear under specific load or timing conditions. Aggregated profiles hide those.
• Race conditions, concurrency bugs, correctness issues. Flamegraphs show where time goes, not why behavior is wrong.

Pair flamegraphs with other tools. Use distributed traces for individual request latency. Use tracepoints/eBPF for rare events. Use metrics for trends. Flamegraphs are the tool for “where is time going right now,” and that’s enough to make them a weekly-use tool for any team doing serious performance work.

The investment in making flamegraphs first-class in your engineering process — continuous profiling, release-to-release diffs, CI integration — pays back every time a performance problem surfaces. You stop starting from scratch; you start from a picture. And most of the time, the picture answers the question before you finish asking it.