BPF Performance Tools Tour: bcc, bpftrace, and libbpf
There’s a decent chance that the next time you diagnose a nasty Linux performance problem, the tool that actually cracks it will be built on eBPF. The ability to attach safe, JIT-compiled programs to kernel tracepoints, kprobes, uprobes, and scheduling events — and collect per-event data without noticeably perturbing the system — has reshaped Linux observability over the last decade. strace was the old “attach to a running process and see what’s happening”; eBPF is the modern equivalent that works at production scale.
This post is a tour of the BPF performance tools ecosystem: bcc, bpftrace, and libbpf — what each is for, when to reach for which, and a practical catalog of the tools you should know. I’ll focus on ready-to-run tools rather than teaching eBPF programming end-to-end (that’s a longer project), but I’ll show enough to get you writing one-liners that answer real questions.
The ecosystem in one paragraph
eBPF itself is the kernel mechanism: a virtual machine inside the kernel that runs verified, sandboxed programs attached to events. The tooling on top:
- bcc (BPF Compiler Collection) — Python + C library for building BPF tools. Most of the classic tools (
execsnoop,biolatency,tcpconnect) started here. Tools are Python scripts that compile C code at runtime and load it into the kernel. - bpftrace — a DSL for ad-hoc BPF. DTrace-inspired syntax. Write one-liners like
bpftrace -e 'kprobe:vfs_read { @[comm] = count(); }'and get results instantly. - libbpf / CO-RE — the modern approach. BPF programs written in C, compiled once with CO-RE (Compile Once, Run Everywhere), distributed as small self-contained binaries. The BCC tools have been gradually rewritten as libbpf-tools for this reason.
- bpftop, bpftool — utilities for inspecting and managing loaded BPF programs.
For the tool user rather than the tool author, the short version: bpftrace for ad-hoc investigation, bcc / libbpf-tools for production-grade canned tools. You don’t need to know much about eBPF to use them effectively.
Installing and privileges
On recent distros:
|
|
The bpfcc-tools package installs classic bcc tools with a -bpfcc suffix (execsnoop-bpfcc). The libbpf-tools suite is distributed separately on some distros or built from source.
Permissions: most BPF operations require CAP_BPF plus some combination of CAP_PERFMON, CAP_NET_ADMIN, and CAP_SYS_ADMIN depending on what you’re probing. In practice, most tools require root. For unprivileged eBPF there’s a long kernel roadmap but it’s not yet broadly useful.
Kernel version matters. Modern eBPF features (CO-RE, ring buffers, BTF) need kernel 5.4+. Everything substantial works on 5.10+. Current LTS (6.1, 6.6, 6.12) is the sweet spot.
bpftrace: the DTrace-alike
bpftrace is the tool for ad-hoc investigations. The syntax is a small DSL: attach point, filter, action.
probe_type:probe_name /filter/ { action }
A few canonical examples:
|
|
The attach points:
tracepoint:<subsystem>:<event>— static kernel tracepoints. Stable across kernel versions.bpftrace -l 'tracepoint:*'lists thousands of them.kprobe:<function>— dynamic: attach to any kernel function entry. Fast to set up, unstable across kernel versions (functions get renamed).kretprobe:<function>— kernel function return. Useful for measuring function duration.uprobe:<binary>:<function>— userspace function entry. Works on any binary with symbols.uretprobe:<binary>:<function>— userspace function return.profile:hz:99— timer-based sampling. For stack-based profiling at 99 Hz.interval:s:10— fires every N seconds. For periodic summarization.
Built-in variables:
pid— current PID.tid— current thread ID.comm— command name.nsecs— nanoseconds since boot.cpu— current CPU.args— tracepoint arguments.arg0,arg1, … — kprobe/uprobe arguments.
Aggregation built-ins (the killer feature):
count()— counter.sum(expr),avg(expr),min(expr),max(expr)— scalar aggregations.hist(expr)— log2 histogram.lhist(expr, min, max, step)— linear histogram with explicit bounds.
The hist primitive is what makes bpftrace so useful. Instead of dumping every event to a file and post-processing, the kernel aggregates into a histogram on the fly and prints the result when the script exits.
@us:
[1K, 2K) 12 | |
[2K, 4K) 187 |@@ |
[4K, 8K) 1832 |@@@@@@@@@@@@@@@@@@@@ |
[8K, 16K) 4598 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16K, 32K) 1203 |@@@@@@@@@@@@@ |
[32K, 64K) 301 |@@@ |
This is the p50/p95/p99 picture of disk read latency (in microseconds) on a running system, with overhead typically under 1%. Try producing that with strace + awk.
bpftrace scripts vs one-liners
bpftrace one-liners are for exploration. When a pattern crystallizes, save it as a .bt file:
#!/usr/bin/env bpftrace
BEGIN {
printf("Tracing slow reads > 10ms. Ctrl-C to end.\n");
}
kprobe:vfs_read {
@start[tid] = nsecs;
}
kretprobe:vfs_read /@start[tid]/ {
$elapsed = (nsecs - @start[tid]) / 1000000;
if ($elapsed > 10) {
printf("%s (%d) read took %d ms\n", comm, pid, $elapsed);
}
delete(@start[tid]);
}
Saves the tracking state per-thread, computes elapsed milliseconds, filters for > 10 ms. Many of the “BPF Performance Tools” book’s examples follow this shape.
The bcc-tools catalog
BCC ships with ~150 ready-made tools. You can’t remember them all; the trick is knowing what categories exist. Here’s a working mental map:
Process tracing:
execsnoop— everyexec()system-wide. Catches command-line tools being run. Indispensable for finding what’s running on a host.exitsnoop— every process exit with its return code.killsnoop— everykill()syscall. Finds the culprit when a process is being killed.opensnoop— every file open. Shows which processes are touching which files.
Storage / filesystem:
biolatency— block I/O latency histogram. First thing to run when disks are suspect.biosnoop— per-I/O trace with latency.biotop— top-like view of per-process disk I/O.ext4slower,xfsslower,btrfsslower— filesystem operations slower than N ms.filelife— short-lived file creates/deletes. Finds processes churning tempfiles.filetop— top-like view of files being read/written per process.
Networking:
tcpconnect— every outbound TCP connection.tcpaccept— every inbound TCP accept.tcpretrans— TCP retransmissions. Finds flaky links.tcpconnlat— TCP connection latency histogram.tcptop— top-like TCP throughput per-connection.tcplife— per-session duration and bytes sent.sockmetrics— socket-level stats.
Memory / page cache:
cachestat— page cache hit rate. Shows if the cache is effective.cachetop— per-process cache stats.memleak— allocation tracer that finds unfreed allocations. Slow but powerful.slabratetop— slab allocator churn by cache.
Scheduling:
runqlat— histogram of how long threads wait in the run queue. Shows scheduler saturation.runqlen— run queue length.runqslower— processes waiting longer than N ms for CPU.offcputime— stack-based profile of why threads are off-CPU (blocked). Prints the raw data for flamegraphs.cpudist— on-CPU time histogram per process.
Syscalls:
syscount— syscalls per second per process.statsnoop— everystat().argdist— arbitrary function argument distribution.funccount— function call counts matching a pattern.funclatency— function latency histogram.funcslower— function calls slower than N ms.
Language-specific:
uflow— Python/Java/Ruby method call flow graph.ustat— Python/Java/Ruby runtime stats (allocations, GC events).uthreads— thread start/stop events.ugc— GC events.dbslower,dbstat— MySQL/Postgres query latency.
Every one of these is a focused tool that answers a specific question. You don’t need to know how they work internally; you need to know they exist.
A diagnostic session using BPF tools
Concrete example: a service’s P99 latency doubled. Here’s how I’d use BPF tools:
|
|
In 10 minutes of focused tool runs, you’ve narrowed the problem from “service is slow” to a specific saturated resource or contested lock. That’s the eBPF promise: every question has a cheap, targeted answer.
Ring buffers and modern eBPF
Classic BCC tools use perf buffers to ship data from the kernel to userspace. Each CPU has its own buffer; events can be lost if a buffer fills. Modern eBPF uses the BPF ring buffer (added in kernel 5.8), which is a single MPMC queue with better event ordering and lower overhead.
For tools built on libbpf with CO-RE, ring buffers are the default. If you’re writing a new BPF tool today, use ring buffers. For existing BCC tools, the difference usually doesn’t matter to the user — they produce the same output.
libbpf-tools: the next-gen canned tools
The BCC Python-based approach has drawbacks:
- Each tool compiles its C code at runtime (heavy).
- Python + LLVM dependencies are substantial.
- Tools embed kernel assumptions that break on kernel upgrades.
libbpf-tools is a reimplementation of the BCC tools as standalone C programs using libbpf and CO-RE. Benefits:
- Single binary, no runtime compilation, small (tens of KB).
- Works across kernel versions via BTF (BPF Type Format) field relocation.
- Faster startup, lower overhead.
Many distros ship both. The -libbpf suffix distinguishes them:
|
|
Behavior is largely identical. For production monitoring, libbpf-tools are the better choice — smaller footprint, more reliable across kernel upgrades. For active development and quick iteration on new tools, bcc and bpftrace are still faster.
Writing your own BPF tools
Sometimes no canned tool answers your exact question. The options:
bpftrace for ad-hoc: a .bt script is the fastest way to answer a specific one-time question. Write, run, capture results, throw away.
libbpf + C for production: for tools you’ll ship, monitor with, or include in a product. Write BPF programs in C, compile with clang -target bpf, load with libbpf. Substantial learning curve but low runtime overhead and excellent portability.
BCC (Python + C) for middle-ground: scripts you’ll reuse, want to iterate on, but don’t need to be minimum-overhead. Python glue plus embedded BPF C code.
Go / Rust libraries: cilium/ebpf (Go) and aya (Rust) are mature alternatives to libbpf for people who’d rather not write C glue.
For learning the libbpf CO-RE path, the kernel’s samples/bpf/ and tools/testing/selftests/bpf/ are useful examples. Andrii Nakryiko’s blog posts on CO-RE are the best walkthrough material.
A minimal libbpf-tools-style program has two files:
tool.bpf.c— the BPF program, compiled for the BPF virtual machine.tool.c— the userspace loader, compiled normally.
A build pipeline compiles the BPF program, embeds it as a blob in the userspace binary, and links against libbpf. The result is a single small binary you can ship.
Continuous profiling with BPF
Parca (Polar Signals), Grafana Pyroscope (eBPF mode), Coroot, and Elastic Universal Profiler all use eBPF for continuous CPU profiling with no language-specific agents. The approach:
- Sample stack traces at 19–99 Hz across all processes using a BPF
profile:hz:19probe. - Unwind stacks in the kernel (using DWARF or frame pointers).
- Ship aggregated profiles to a backend.
- Render as flamegraphs, queryable by time range.
This is the current frontier of production profiling. You install one agent per host, and suddenly every service has a flamegraph, continuously, with minimal overhead. For teams that want continuous profiling without touching application code, eBPF profilers are the lowest-friction path.
The tradeoff vs language-native profilers (async-profiler, py-spy): BPF profilers don’t know about JIT-ed code unless the runtime emits perf maps. For JVMs, Python, Node, V8 — you may see hex addresses instead of function names at the top of stacks. The better BPF profilers (Parca’s rbperf, Pyroscope’s language-aware mode) walk language-specific data structures to recover symbols.
Networking with BPF: beyond observability
eBPF isn’t just observation. Its biggest production deployment is in networking:
- Cilium — Kubernetes CNI built entirely on eBPF. Load balancing, network policy, service mesh — all via BPF programs attached at the TC (traffic control) layer and XDP (eXpress Data Path).
- Katran — Facebook’s L4 load balancer, XDP-based. Handles absurd PPS rates.
- tc / XDP — the kernel’s data-path attachment points. XDP runs before the kernel networking stack — the earliest possible hook. Used for DDoS mitigation, load balancing, packet filtering at line rate.
Tools like tcplife, tcpretrans cross-pollinate — the same BPF hooks that enable Cilium’s data plane also enable the tooling that tells you exactly what’s happening at the network layer on your host.
Security and BPF
Complementary to performance tools:
- Tetragon — Isovalent’s runtime security engine built on eBPF. Policy-based process and network observability at the kernel level.
- Falco — CNCF project; has traditional kernel module and a newer eBPF-based driver. Alerts on syscall-level security events.
- BPF LSM — the kernel’s Linux Security Module framework now supports eBPF LSM programs, letting you write security policies in BPF.
The security and performance toolchains share infrastructure. Many of the probes used by Falco (tracepoint on syscall entry) are the same ones you’d use in a bpftrace one-liner.
Common gotchas
A few things that trip people up:
Missing kernel headers or BTF. Some distros don’t ship /sys/kernel/btf/vmlinux. Without it, CO-RE doesn’t work; you need to install kernel-debuginfo or use the older header-based approach. Recent Ubuntu/Fedora/RHEL (9+) include BTF by default.
High-overhead probes on hot paths. Attaching a kprobe to a function that runs millions of times per second will slow the system noticeably. Check /proc/interrupts and system-wide CPU while the probe is active. If overhead is too high, move to tracepoints (cheaper) or sampling (profile:hz:99).
Missing stack frames. Without frame pointers, BPF stack walking via kstack/ustack is incomplete. Use --dwarf where supported or recompile binaries with -fno-omit-frame-pointer.
Tracepoint vs kprobe compatibility. Kprobes break across kernel upgrades because function names and signatures change. Prefer tracepoints when they exist — they’re a stable ABI.
Per-CPU data races. BPF maps are per-CPU or global; choosing wrong causes either memory blowup (global with high cardinality) or lost updates (per-CPU hash with shared keys). Understand what kind of map you’re using.
Output truncation. bpftrace truncates long strings and stacks by default. Increase BPFTRACE_STRLEN and ustack(N) depth as needed.
A catalog worth internalizing
If you take one thing from this post, internalize this shortlist. Next time you’re in a performance investigation on Linux, run one of these first:
execsnoop— “what’s running on this host?”opensnoop— “what files are being touched?”biolatency— “is disk slow?”tcplife— “what TCP connections happened?”tcpretrans— “is the network lossy?”runqlat— “is the scheduler saturated?”offcputime— “what’s blocking?”cachestat— “is the page cache working?”funclatency— “how slow is this kernel function?”profile— “sample CPU, build a flamegraph.”
Plus bpftrace for anything ad-hoc. Between these, you can answer most of the questions that arise in Linux performance work without installing anything else.
Where BPF fits in 2026
BPF tools haven’t replaced perf, tracepoints, ftrace, or language-specific profilers. They’ve joined them. The decision tree I use:
- Performance counters (IPC, cache misses) →
perf stat. - CPU-bound, “where is time spent” →
perf record+ flamegraph, or BPF profiler. - Latency issues (“why is this slow”) →
bpftrace/ BCC / libbpf-tools to pinpoint. - Application-level detail → language-native profiler (async-profiler, py-spy, pprof).
- Continuous/production → eBPF-based profiler (Parca, Pyroscope).
- Security / audit → eBPF-based security tool (Tetragon, Falco).
The common thread: eBPF isn’t a replacement for any one of these. It’s the substrate that makes many of them production-viable. The days when deep Linux introspection required kernel modules, DTrace ports, or heavyweight agents are past. BPF tools give you DTrace-quality answers with a Ctrl-C-able lifetime and negligible overhead, and they work on every modern Linux host.
The learning curve is real — bpftrace syntax takes a day to get comfortable with, libbpf-tools a week. But the return on investment, measured in hours saved per incident, is enormous. Every senior Linux engineer should have these tools at their fingertips. Most of the hard performance problems of the last decade got easier because of them.
Comments