eBPF for Observability: Writing Your First Program

For most of Linux’s history, instrumenting the kernel meant two bad choices: add printk statements and recompile, or use SystemTap/DTrace-style tools that required kernel modules with no safety guarantees and a good chance of crashing your machine if you got something wrong. Neither was something you did casually on a production box.

eBPF changed that. It is a technology built into the Linux kernel — no modules, no recompilation — that lets you attach small, verified programs to hundreds of kernel and userspace hooks. The programs run at JIT-compiled native speed. The verifier statically checks them before they run and rejects anything that could crash the kernel or loop forever. You can attach a tracer to a live production system, collect the data you need, and detach without ever restarting a process or risking stability.

This post starts with how eBPF works internally (the parts that matter for using it effectively), moves through bpftrace as the fastest path to immediate insight, then covers writing a real program in C with libbpf, and finishes with the production tools — Cilium, Pixie, Parca — that have made eBPF the foundation of modern cloud-native observability.

What eBPF Is, Precisely

The name is a historical artifact: “extended Berkeley Packet Filter.” The original BPF was a packet filter mechanism from 1992. The “extended” version, introduced in Linux 3.18 (2014) and substantially expanded through 4.x and 5.x, is a general-purpose in-kernel execution engine with almost nothing in common with its predecessor beyond the name.

An eBPF program is a small chunk of bytecode — a restricted instruction set with 64-bit registers — that you load into the kernel via the bpf(2) system call. The kernel runs it through a verifier and a JIT compiler, then attaches it to a hook point. When that hook fires (a syscall, a kernel function entry, a network packet arriving), your program runs in kernel context.

The key properties:

• Verified safety: The verifier performs static analysis before the program runs. It checks that all memory accesses are bounds-checked, there are no null pointer dereferences, all branches eventually terminate (bounded loops only), and the program does not call arbitrary kernel functions (only a curated set of BPF helper functions). A program that fails verification is rejected — it never runs.
• JIT compiled: After verification, the bytecode is compiled to native machine code. There is no interpreter overhead. An eBPF program runs as fast as equivalent kernel code.
• No kernel recompilation: Programs are loaded at runtime into the running kernel. You can attach and detach them without rebooting, without kernel modules, without touching any running process.
• Data sharing via maps: eBPF programs communicate with userspace through BPF maps — typed, kernel-managed data structures (hash tables, arrays, ring buffers, LRU caches) that both kernel-side eBPF code and userspace programs can read and write.

Userspace                           Kernel
─────────────────────────────────────────────────────
                    bpf() syscall
Your program ──────────────────────►  Verifier
                                        │ static analysis
                                        │ safety checks
                                        ▼
                                      JIT compiler
                                        │ bytecode → native
                                        ▼
                                      Attach to hook
                                      ┌──────────────────────┐
                                      │  Hook fires (event)  │
                                      │  eBPF program runs   │
                                      │  Writes to BPF map   │
                                      └──────────────────────┘
Your program ◄──────────────────────  Read BPF map
                    mmap / read

Attachment Points: Where Programs Run

eBPF programs attach to hooks. The hook determines what data is available (which kernel structs you can read), what actions you can take, and what privileges are required. For observability work, the relevant categories are:

For observability, you will almost always start with tracepoint (stable ABI, kernel-guaranteed arguments) and fall back to kprobe when the tracepoint does not exist or does not expose the data you need. perf_event at 99 Hz is the standard approach for CPU flame graphs.

bpftrace: Immediate Insight Without Writing C

bpftrace is a high-level tracing language built on eBPF. Its syntax resembles awk: probe /filter/ { action }. It compiles the script to eBPF bytecode, loads it, and prints output — all in one command. It is the right tool for ad hoc investigation, one-off profiling, and learning the eBPF model before writing programs in C.

Install it:

1
2
3
4
5

# Ubuntu/Debian
sudo apt install bpftrace

# Verify kernel and BTF support
bpftrace --info | grep -E "BTF|kernel"

Essential one-liners

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

# What files are being opened, and by whom?
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

# How many syscalls per process?
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# Latency distribution of read() calls (histogram in microseconds)
sudo bpftrace -e '
  tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }
  tracepoint:syscalls:sys_exit_read  /@start[tid]/ {
    @us = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
  }'

# Block I/O latency by device
sudo bpftrace -e '
  tracepoint:block:block_rq_issue   { @start[args->dev, args->sector] = nsecs; }
  tracepoint:block:block_rq_complete /@start[args->dev, args->sector]/ {
    @ms[args->dev] = hist((nsecs - @start[args->dev, args->sector]) / 1000000);
    delete(@start[args->dev, args->sector]);
  }'

# TCP retransmissions with source/dest
sudo bpftrace -e '
  tracepoint:tcp:tcp_retransmit_skb {
    printf("%-15s %-6d → %-15s %-6d %s\n",
      ntop(args->saddr), args->sport,
      ntop(args->daddr), args->dport,
      comm);
  }'

# CPU profiling: on-CPU kernel stacks at 99 Hz (for flame graphs)
sudo bpftrace -e 'profile:hz:99 { @[kstack] = count(); }' -c 30 > stacks.txt

# Who is calling malloc and how large are the allocations?
sudo bpftrace -e '
  uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc { @allocs[comm] = hist(arg0); }'

# OOM killer activations
sudo bpftrace -e '
  kprobe:oom_kill_process {
    printf("OOM kill: %s (pid %d)\n", comm, pid);
    print(kstack);
  }'

# DNS queries (traces getaddrinfo calls in libc)
sudo bpftrace -e '
  uprobe:/lib/x86_64-linux-gnu/libc.so.6:getaddrinfo {
    printf("%s queried: %s\n", comm, str(arg0));
  }'

The @ prefix creates a BPF map. @[key] = count() is a frequency counter keyed by the expression in brackets. hist() produces a log-scale histogram. All maps are printed at exit (or Ctrl-C). This is the bpftrace mental model: attach hooks that write to maps, read the maps.

Writing a short script

One-liners get unwieldy for multi-probe analysis. A script file:

#!/usr/bin/env bpftrace
// Track slow disk reads: reads that take longer than 10ms

tracepoint:block:block_rq_issue /args->rwbs == "R"/ {
    @issue_time[args->dev, args->sector] = nsecs;
}

tracepoint:block:block_rq_complete {
    $key_dev = args->dev;
    $key_sec = args->sector;
    if (@issue_time[$key_dev, $key_sec]) {
        $latency_ms = (nsecs - @issue_time[$key_dev, $key_sec]) / 1000000;
        if ($latency_ms > 10) {
            printf("SLOW READ: dev=%d latency=%dms bytes=%d\n",
                   args->dev, $latency_ms, args->nr_sector * 512);
        }
        delete(@issue_time[$key_dev, $key_sec]);
    }
}

END { clear(@issue_time); }

1

sudo bpftrace slow_reads.bt

Writing a Real Program: libbpf and C

bpftrace is great for investigation. For a persistent tracer, a daemon that records metrics, or anything that needs to process data in userspace beyond what a bpftrace script can do, you write the eBPF program in C and load it with libbpf.

The modern approach is CO-RE (Compile Once, Run Everywhere). Instead of compiling against your specific kernel’s header files (which means your binary only works on that kernel version), you compile against a vmlinux.h generated from BTF (BPF Type Format) data. BTF encodes kernel struct layouts. At load time, libbpf reads the running kernel’s BTF and patches any struct field offsets that changed between the kernel you compiled on and the kernel you’re running on. One binary, many kernel versions.

Setup

1
2
3
4
5

# Install dependencies
sudo apt install clang llvm libbpf-dev linux-tools-$(uname -r) linux-headers-$(uname -r)

# Generate vmlinux.h from running kernel's BTF
bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

The eBPF C program (kernel side)

This tracer records every execve() call with the process name, PID, and the command being executed — useful for tracking what processes are spawning on a busy system:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53

// execsnoop.bpf.c  — the kernel-side program
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>
#include <bpf/bpf_tracing.h>

#define TASK_COMM_LEN 16
#define ARGV_LEN      128

// The event struct we'll send to userspace
struct exec_event {
    u32 pid;
    u32 ppid;
    char comm[TASK_COMM_LEN];   // parent process name
    char filename[ARGV_LEN];    // executable being launched
};

// Ring buffer map — low-overhead, ordered, per-CPU
struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);  // 256KB ring buffer
} events SEC(".maps");

// Attach to the sys_enter_execve tracepoint
SEC("tracepoint/syscalls/sys_enter_execve")
int trace_execve(struct trace_event_raw_sys_enter *ctx)
{
    struct exec_event *e;
    struct task_struct *task;

    // Reserve space in the ring buffer
    e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
    if (!e)
        return 0;

    // Populate the event
    e->pid  = bpf_get_current_pid_tgid() >> 32;
    task    = (struct task_struct *)bpf_get_current_task();
    e->ppid = BPF_CORE_READ(task, real_parent, tgid);

    bpf_get_current_comm(e->comm, sizeof(e->comm));

    // Read the filename argument from userspace memory
    // ctx->args[0] is the first argument to execve (const char *filename)
    const char *filename = (const char *)ctx->args[0];
    bpf_probe_read_user_str(e->filename, sizeof(e->filename), filename);

    // Submit to ring buffer
    bpf_ringbuf_submit(e, 0);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

The userspace loader (C)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

// execsnoop.c — userspace loader and event consumer
#include <stdio.h>
#include <signal.h>
#include <bpf/libbpf.h>
#include "execsnoop.skel.h"   // auto-generated by bpftool gen skeleton

static volatile bool exiting = false;

static void sig_handler(int sig) { exiting = true; }

// Callback invoked for each event in the ring buffer
static int handle_event(void *ctx, void *data, size_t size)
{
    struct exec_event *e = data;
    printf("%-8d %-8d %-16s %s\n",
           e->pid, e->ppid, e->comm, e->filename);
    return 0;
}

int main(void)
{
    struct execsnoop_bpf *skel;
    struct ring_buffer    *rb;

    // Open, load, and verify the BPF program
    skel = execsnoop_bpf__open_and_load();
    if (!skel) {
        fprintf(stderr, "Failed to open BPF skeleton\n");
        return 1;
    }

    // Attach to the tracepoint
    if (execsnoop_bpf__attach(skel)) {
        fprintf(stderr, "Failed to attach\n");
        goto cleanup;
    }

    // Set up ring buffer polling
    rb = ring_buffer__new(bpf_map__fd(skel->maps.events), handle_event, NULL, NULL);

    signal(SIGINT, sig_handler);
    signal(SIGTERM, sig_handler);

    printf("%-8s %-8s %-16s %s\n", "PID", "PPID", "PARENT", "EXEC");
    printf("%-8s %-8s %-16s %s\n", "---", "----", "------", "----");

    while (!exiting) {
        ring_buffer__poll(rb, 100 /* timeout_ms */);
    }

cleanup:
    ring_buffer__free(rb);
    execsnoop_bpf__destroy(skel);
    return 0;
}

Build

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# Makefile
CLANG   = clang
BPFTOOL = bpftool

all: execsnoop

# Compile the eBPF C to BPF bytecode
execsnoop.bpf.o: execsnoop.bpf.c vmlinux.h
	$(CLANG) -g -O2 -target bpf -D__TARGET_ARCH_x86 \
	         -I/usr/include/bpf \
	         -c execsnoop.bpf.c -o execsnoop.bpf.o

# Generate the skeleton header from the compiled object
execsnoop.skel.h: execsnoop.bpf.o
	$(BPFTOOL) gen skeleton execsnoop.bpf.o > execsnoop.skel.h

# Compile the userspace loader
execsnoop: execsnoop.c execsnoop.skel.h
	$(CC) -g -O2 execsnoop.c -o execsnoop -lbpf -lelf -lz

clean:
	rm -f execsnoop execsnoop.bpf.o execsnoop.skel.h

vmlinux.h:
	$(BPFTOOL) btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

1
2

make
sudo ./execsnoop

Output:

PID      PPID     PARENT           EXEC
---      ----     ------           ----
184321   184320   bash             /usr/bin/ls
184322   1        systemd          /usr/lib/postfix/sbin/pickup
184325   184321   ls               /usr/bin/grep

This runs with zero impact on the processes being traced. There is no ptrace, no LD_PRELOAD, no signal sent to any process. The kernel fires the eBPF program at the tracepoint hook and your data appears in the ring buffer.

Key BPF Map Types

Maps are how eBPF programs store state and communicate with userspace. Choosing the right map type matters for both correctness and performance:

For event streaming, BPF_MAP_TYPE_RINGBUF is the right default in kernels 5.8+. It is more memory-efficient than PERF_EVENT_ARRAY (one buffer shared across all CPUs rather than one per CPU), preserves event ordering, and supports reserve/commit/discard patterns that allow writing complex events without intermediate copies.

Production Tools Built on eBPF

Writing eBPF programs in C is something you do when you have a specific measurement need that no existing tool covers. For common observability tasks, purpose-built tools handle the complexity for you.

Cilium and Hubble

Cilium replaces kube-proxy and iptables-based networking in Kubernetes with eBPF. Every packet path — service load balancing, network policy enforcement, connection tracking — runs through eBPF programs attached to XDP and TC hooks. The performance consequence is significant: iptables at scale involves traversing long rule chains per packet; eBPF policies are constant-time regardless of how many rules are defined.

Hubble is Cilium’s observability layer. It instruments the same eBPF hooks to produce L3/L4/L7 flow records — every TCP connection, every HTTP request, every DNS query, attributed to source and destination pod. No application instrumentation required. No sidecar proxy.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Install Cilium with Hubble
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

# View live network flows
hubble observe --follow --namespace default

# See all HTTP requests from a specific pod
hubble observe --pod frontend/frontend-abc123 --protocol http

# View a network policy verdicts (dropped packets)
hubble observe --verdict DROPPED

Hubble’s network flow data is the fastest way to answer “why can’t pod A reach pod B” in a Kubernetes cluster — it shows you the exact packet flow and the policy verdict (FORWARDED/DROPPED) at every hop.

Pixie

Pixie is zero-instrumentation application observability for Kubernetes. It uses uprobes attached to Go, Python, JVM, and Node.js runtimes to capture application-level protocol data without any code changes and without sidecar proxies. HTTP/1.1, HTTP/2, gRPC, DNS, PostgreSQL, MySQL, Redis, Kafka — Pixie captures request/response pairs for all of these directly from kernel memory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Install Pixie
px deploy

# Query HTTP request latencies for a service
px run px/http_data -- -start_time '-10m' -namespace 'default'

# Get a flame graph for a specific pod
px run px/cpu_flamegraph -- -pod 'default/myservice-abc123'

# Database query performance
px run px/mysql_data -- -start_time '-5m'

Pixie’s query language (PxL, Python-like) lets you write custom analyses against the data Pixie is continuously collecting. Because the collection is kernel-native via eBPF, the overhead is typically 2-5% CPU compared to 10-30% for agent-based collection.

Parca: Continuous Profiling

Parca is a continuous profiling system. It samples stack traces across all processes on a host at 19 samples per second per CPU using a perf_event eBPF program, stores the aggregated profiles, and serves them as a flamegraph UI. Unlike profiling you turn on for a specific incident, Parca runs all the time — so when you have a CPU spike, the profile data is already there.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# parca.yaml — basic configuration
object_storage:
  bucket:
    type: FILESYSTEM
    config:
      directory: ./data

scrape_configs:
  - job_name: default
    scrape_interval: 10s
    targets:
      - targets: ['localhost:7100']   # parca-agent on each host

1
2
3
4

# Run the agent on each host
parca-agent --node=$(hostname) --log-level=info

# Parca stores profiles and serves them at localhost:7070

The flame graph view in Parca shows CPU time broken down by stack frame, continuously over time. You can select any time window and see where CPU was spent — without having added any profiling hooks to your applications.

bcc Tools: The Swiss Army Knife

The BCC (BPF Compiler Collection) project maintains a library of ready-to-run eBPF tools for specific performance questions. Most Linux distributions package them:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

sudo apt install bpfcc-tools linux-headers-$(uname -r)

# Open files, with latency
sudo opensnoop-bpfcc

# TCP connection events
sudo tcpconnect-bpfcc
sudo tcpaccept-bpfcc

# Syscall count by process
sudo syscount-bpfcc -p $(pgrep nginx)

# Disk I/O latency histogram
sudo biolatency-bpfcc

# Who's hitting swap?
sudo swapin-bpfcc

# Off-CPU time (time processes are blocked)
sudo offcputime-bpfcc -p $(pgrep postgres) -f 30 > offcpu.txt
flamegraph.pl --color=io --title="Off-CPU" offcpu.txt > offcpu.svg

# Block I/O by process
sudo biotop-bpfcc

# DNS query latency
sudo dnssnoop-bpfcc

These are production-safe tools. The BCC repository (github.com/iovisor/bcc) has over 100 of them, each solving a specific measurement problem that previously required either invasive instrumentation or guesswork.

Kernel Requirements

eBPF capabilities have accumulated steadily across kernel versions. The minimum practical baseline for observability work is kernel 4.18 (2018), but several important features require newer kernels:

Most modern Linux distributions running in production (Ubuntu 20.04+, Debian 11+, RHEL 8+, any kernel from 2020 onward) are comfortably in the 5.x range and support the full modern eBPF feature set. Check with uname -r before reaching for CO-RE features.

The Verifier: Why eBPF Programs Get Rejected

Understanding the verifier helps you debug the “permission denied” and “invalid program” errors you will inevitably encounter when writing eBPF programs.

The verifier performs a simulated execution of every possible path through your program. It tracks the type and range of values in each register. At every memory access, it checks that the pointer is valid and the offset is in bounds. It rejects programs that:

• Access memory through an unverified pointer (pointer not obtained from a known-valid source)
• Perform arithmetic that could produce an out-of-bounds pointer
• Call kernel functions not in the allowed helper set
• Have any code path that could loop unboundedly
• Exceed the instruction limit (1 million instructions as of kernel 5.2)
• Read uninitialized stack memory

Common rejection patterns:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

// REJECTED: pointer arithmetic without range check
u32 index = some_value;
u64 *ptr = array + index;   // verifier: index could be out of bounds
*ptr = 1;

// ACCEPTED: range-checked first
u32 index = some_value;
if (index >= MAX_ENTRIES) return 0;
u64 *ptr = array + index;   // verifier: index is proven < MAX_ENTRIES
*ptr = 1;

// REJECTED: reading struct field without BPF_CORE_READ
struct task_struct *task = bpf_get_current_task();
u32 pid = task->pid;   // NOT safe — direct dereference of kernel pointer

// ACCEPTED: use the helper
struct task_struct *task = bpf_get_current_task();
u32 pid = BPF_CORE_READ(task, pid);   // safe

The BPF_CORE_READ macro expands to bpf_probe_read_kernel() internally, which the verifier understands as a safe kernel memory read. Direct pointer dereferences of kernel pointers are rejected because the verifier cannot prove the pointer is still valid at the point of access.

When to Reach for eBPF

eBPF is genuinely powerful, but it is not always the right tool. A practical assessment:

Reach for bpftrace when: you have a production performance problem right now and need to answer “what is this process actually doing” within minutes. It is the fastest path from question to data.

Reach for libbpf/C when: you need a persistent tracer, a daemon that exports metrics to Prometheus, or custom logic that processes events in userspace beyond what bpftrace supports. The development cycle is longer but the result is deployable infrastructure.

Use the existing tools first: If you need syscall-level tracing, bcc’s opensnoop, tcpconnect, and biolatency exist and are battle-tested. If you’re on Kubernetes and need network visibility, Cilium Hubble covers most of what you’d otherwise write a custom tracer for. If you need continuous CPU profiling without instrumentation, deploy Parca. Building custom eBPF programs makes sense when you have a specific measurement requirement these tools do not address.

What eBPF does not do: it does not instrument application logic (for that, you want OpenTelemetry SDK tracing), it does not give you business-level metrics (order counts, user actions), and it does not replace distributed tracing across service boundaries. eBPF observability is kernel-layer and system-layer. It answers “what is the OS doing on behalf of this process” — not “what is the application’s business logic doing.”

The practical superpower is for SRE and platform engineering work: diagnosing latency spikes without application deploys, attributing CPU and I/O to specific processes without invasive instrumentation, and building security monitoring that cannot be bypassed by a compromised application. For those use cases, nothing else is close.