LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

eBPF for Observability

15 min read DevOps
ebpfobservabilitylinuxtracingbpftraceciliumnetworkingperformance
Contents

eBPF is one of the most significant additions to the Linux kernel in the past decade. It lets you run sandboxed programs inside the kernel itself — attaching them to virtually any system event — without modifying kernel source code, loading a kernel module, or rebooting. The result is observability that was previously impossible or required invasive instrumentation: you can trace every system call, follow a network packet from NIC to socket, profile CPU usage at function-call granularity, and measure I/O latency histograms — all in production, with near-zero overhead.

This guide covers the eBPF landscape from first principles through practical tools: what eBPF is and how it works, the BCC toolkit for quick investigations, bpftrace for scriptable tracing, and Cilium for Kubernetes network observability.

What eBPF Is (and How It Works)

The name comes from “extended Berkeley Packet Filter” — the original BPF was a simple bytecode VM for filtering network packets in the kernel (used by tcpdump). Modern eBPF is a complete, general-purpose in-kernel execution environment with almost nothing in common with the original except the name.

Here’s how an eBPF program works:

  1. Write: You write an eBPF program in restricted C (or use a higher-level frontend like bpftrace).
  2. Compile: The compiler generates eBPF bytecode.
  3. Verify: The kernel’s verifier checks the bytecode — it proves the program terminates, doesn’t dereference bad pointers, doesn’t access unauthorized memory, and doesn’t call unsafe functions. If verification fails, the program is rejected.
  4. JIT compile: The kernel JIT-compiles verified bytecode to native machine code.
  5. Attach: The program is attached to a hook point — a kernel event like a system call, a network packet arriving, a kprobe on any kernel function, a tracepoint, or a user-space probe.
  6. Execute: Every time the hook fires, your program runs in kernel context. It can read kernel and user-space data, compute statistics, and write to eBPF maps.
  7. Communicate: eBPF maps are key-value stores shared between the eBPF program (kernel space) and a user-space controller. This is how metrics, traces, and events flow to your observability tooling.

The verifier is what makes eBPF safe. It’s not just “hope the program doesn’t crash the kernel” — it’s a formal proof that the program is safe to run. This is why eBPF can run production observability on live systems that can’t tolerate a kernel panic.

Hook Points

eBPF programs can attach to:

Hook Type What It Traces
kprobe / kretprobe Any kernel function, entry and return
tracepoint Stable kernel trace events (preferred over kprobes)
uprobe / uretprobe Any user-space function in any binary
USDT User-space statically defined tracepoints (in Go, Java, Python, etc.)
XDP Network packets at driver level, before the kernel network stack
TC (Traffic Control) Ingress/egress packets in the network stack
socket filter Packets on a socket
LSM Linux Security Module hooks (MAC policy enforcement)
perf events CPU performance counters, sampling profiler
fentry / fexit Fast alternatives to kprobes (BTF-based, lower overhead)

The eBPF Tooling Ecosystem

There are several layers of tooling:

┌─────────────────────────────────────────┐
│  High-level tools (Cilium, Pixie,       │
│  Parca, Pyroscope, Falco)               │
├─────────────────────────────────────────┤
│  Scripting frontends                    │
│  (bpftrace, kubectl-trace)              │
├─────────────────────────────────────────┤
│  Libraries (BCC, libbpf, cilium/ebpf)  │
├─────────────────────────────────────────┤
│  eBPF bytecode / BPF CO-RE              │
├─────────────────────────────────────────┤
│  Linux kernel eBPF subsystem            │
└─────────────────────────────────────────┘

BCC (BPF Compiler Collection): A toolkit of pre-written eBPF tools for common tasks. Great for immediate investigation — tools like execsnoop, opensnoop, tcptracer, biolatency are single commands.

bpftrace: A high-level tracing language (think awk for eBPF). Write one-liners or short scripts to trace anything. The fastest path from “I need to understand what’s happening” to an answer.

libbpf / BPF CO-RE: A C library for writing eBPF programs that are portable across kernel versions. “CO-RE” (Compile Once – Run Everywhere) uses BTF (BPF Type Format) to relocate struct field offsets at load time rather than requiring kernel headers at compile time.

Cilium: A Kubernetes CNI plugin that uses eBPF for network policy enforcement, load balancing, and deep network observability. Hubble (Cilium’s observability layer) gives you per-flow visibility across your entire cluster.

Prerequisite: Kernel Version and BTF

Most modern eBPF features require a recent kernel. Check yours:

1
2
3
4
5
6
7
8
9

uname -r
# 6.8.0-101-generic — good, anything 5.8+ covers most use cases

# Check if BTF (BPF Type Format) is available — required for CO-RE programs
ls /sys/kernel/btf/vmlinux
# Should exist on kernels built with CONFIG_DEBUG_INFO_BTF=y

# Check available eBPF features
bpftool feature probe

Minimum kernel versions for common features:

  • kprobes: 4.1+
  • tracepoints: 4.7+
  • XDP: 4.8+
  • BTF / CO-RE: 5.2+ (full support 5.8+)
  • fentry/fexit: 5.5+
  • LSM hooks: 5.7+

Ubuntu 22.04 (kernel 5.15) and later ship with everything you need.

BCC Tools: Instant Observability

Install the BCC toolkit:

1
2
3
4
5

# Ubuntu / Debian
sudo apt install bpfcc-tools linux-headers-$(uname -r)

# The tools are installed as name-bpfcc
# e.g., execsnoop-bpfcc, opensnoop-bpfcc

execsnoop — Every New Process

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

sudo execsnoop-bpfcc
# PCOMM          PID    PPID   RET ARGS
# bash           12341  12340    0 /bin/bash
# ls             12342  12341    0 /bin/ls -la
# curl           12343  12341    0 /usr/bin/curl https://example.com

# Filter to a specific UID
sudo execsnoop-bpfcc -u 1000

# Filter to a command name
sudo execsnoop-bpfcc -n curl

Use this to audit what processes a container or service is spawning — useful for detecting unexpected subprocess execution.

opensnoop — Every File Open

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

sudo opensnoop-bpfcc
# PID    COMM               FD ERR PATH
# 1234   nginx               5   0 /etc/nginx/nginx.conf
# 1235   postgres            8   0 /var/lib/postgresql/data/pg_hba.conf

# Filter to a specific process
sudo opensnoop-bpfcc -p 1234

# Show errors only (failed opens)
sudo opensnoop-bpfcc -x
# PID    COMM               FD ERR PATH
# 5678   app                -1   2 /etc/app/missing.conf   # ENOENT

biolatency — Block I/O Latency Histogram

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

sudo biolatency-bpfcc
# Tracing block device I/O... Hit Ctrl-C to end.
# ^C
#      usecs          : count    distribution
#       0 -> 1        : 0        |                    |
#       2 -> 3        : 0        |                    |
#       4 -> 7        : 234      |**                  |
#       8 -> 15       : 1893     |********************|
#      16 -> 31       : 956      |**********          |
#      32 -> 63       : 123      |*                   |
#      64 -> 127      : 45       |                    |
#     128 -> 255      : 12       |                    |

# Per-disk breakdown
sudo biolatency-bpfcc -D

This immediately shows whether your storage is the bottleneck — and where the tail latency lives.

tcptracer — TCP Connection Events

1
2
3
4
5
6

sudo tcptracer-bpfcc
# Tracing TCP established connections. Ctrl-C to end.
# T  PID    COMM             IP SADDR            DADDR            SPORT DPORT
# A  1234   nginx            4  192.168.10.30    192.168.10.1     80    54321  # Accept
# C  5678   curl             4  192.168.10.100   1.2.3.4          54432 443    # Connect
# X  1234   nginx            4  192.168.10.30    192.168.10.1     80    54321  # Close

tcpconnect and tcpaccept

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Who is making outbound TCP connections?
sudo tcpconnect-bpfcc
# PID    COMM             IP SADDR            DADDR            DPORT
# 1234   python3          4  192.168.10.100   10.0.0.1         5432    # DB connection
# 5678   node             4  192.168.10.100   93.184.216.34    443     # HTTPS

# Who is accepting inbound connections?
sudo tcpaccept-bpfcc
# PID    COMM             IP RADDR            RPORT LADDR            LPORT
# 9012   nginx            4  192.168.10.1     54321 192.168.10.30    80

profile — CPU Flame Graph Data

1
2
3
4
5
6
7
8

# Sample CPU stack traces system-wide, 49 Hz, for 30 seconds
sudo profile-bpfcc -F 49 30 > /tmp/stacks.txt

# Convert to flame graph (install FlameGraph first)
git clone https://github.com/brendangregg/FlameGraph
cat /tmp/stacks.txt | \
  ./FlameGraph/stackcollapse-bpfcc.pl | \
  ./FlameGraph/flamegraph.pl > /tmp/flamegraph.svg

The flame graph shows where CPU time is actually spent — much more accurate than top for finding bottlenecks in complex applications.

Other Useful BCC Tools

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Trace system calls for a process
sudo syscount-bpfcc -p 1234

# File system latency histogram
sudo fileslower-bpfcc 10   # show operations slower than 10ms

# Cache hit rate for the page cache
sudo cachestat-bpfcc 1     # refresh every second

# ext4 operation latency
sudo ext4slower-bpfcc 10   # ext4 ops > 10ms

# DNS query tracing (via uprobe on libc getaddrinfo)
sudo gethostlatency-bpfcc

# Malloc/free call tracing (memory leak hunting)
sudo memleak-bpfcc -p 1234

bpftrace: Scripting for Custom Tracing

bpftrace is a high-level language for one-off investigations. Its syntax is inspired by awk and DTrace.

1
2
3
4
5

# Install
sudo apt install bpftrace

# Verify kernel support
sudo bpftrace --info

One-Liners

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

# Count system calls by name system-wide
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_* { @[probe] = count(); }'

# Trace every exec() call with arguments
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve {
  printf("%s called exec: %s\n", comm, str(args->filename));
}'

# Histogram of read() sizes
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read {
  @size_hist = hist(args->count);
}'

# Measure read() latency (entry + return probes)
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_read { @ts[tid] = nsecs; }
tracepoint:syscalls:sys_exit_read  /@ts[tid]/  {
  @latency_us = hist((nsecs - @ts[tid]) / 1000);
  delete(@ts[tid]);
}'

# Who is writing to a specific file?
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_openat /str(args->filename) == "/etc/passwd"/ {
  printf("PID %d (%s) opened /etc/passwd\n", pid, comm);
}'

# TCP connections with latency
sudo bpftrace -e '
kprobe:tcp_connect { @start[tid] = nsecs; }
kretprobe:tcp_connect /@start[tid]/ {
  printf("connect took %d us\n", (nsecs - @start[tid]) / 1000);
  delete(@start[tid]);
}'

# Top 10 processes by system call count (runs for 5 seconds)
sudo bpftrace -e '
tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }
END { print(@, 10); }' -c "sleep 5"

bpftrace Scripts

For anything more complex, write a .bt script:

#!/usr/bin/env bpftrace
// File: slow-reads.bt
// Show read() calls slower than 1ms, with process and file descriptor

tracepoint:syscalls:sys_enter_read
{
    @start[tid] = nsecs;
    @fd[tid]    = args->fd;
}

tracepoint:syscalls:sys_exit_read
/@start[tid]/
{
    $latency_us = (nsecs - @start[tid]) / 1000;
    if ($latency_us > 1000) {
        printf("%-8d %-16s fd=%-4d latency=%d us\n",
               pid, comm, @fd[tid], $latency_us);
    }
    delete(@start[tid]);
    delete(@fd[tid]);
}

1

sudo bpftrace slow-reads.bt

Another example — tracking memory allocations to find leaks:

#!/usr/bin/env bpftrace
// File: malloc-tracker.bt
// Track outstanding malloc calls in a specific process
// Usage: sudo bpftrace malloc-tracker.bt -p <PID>

uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc
{
    @allocs[tid] = nsecs;
    @size[tid]   = arg0;
}

uretprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc
/@allocs[tid]/
{
    @outstanding[retval] = @size[tid];
    delete(@allocs[tid]);
    delete(@size[tid]);
}

uprobe:/lib/x86_64-linux-gnu/libc.so.6:free
{
    delete(@outstanding[arg0]);
}

interval:s:10
{
    printf("\n=== Outstanding allocations (%d) ===\n",
           count(@outstanding));
    print(@outstanding);
}

Tracing User-Space Functions (uprobes)

bpftrace can trace any function in any binary:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Trace Go's runtime.mallocgc (memory allocation)
sudo bpftrace -e '
uprobe:/usr/local/bin/myapp:runtime.mallocgc {
  @alloc_sizes = hist(arg0);
}'

# Trace Python function calls (requires USDT probes)
sudo bpftrace -e '
usdt:/usr/bin/python3:python:function__entry {
  printf("%s:%s\n", str(arg0), str(arg1));
}'

# Trace a specific function in your own binary
sudo bpftrace -e '
uprobe:/usr/local/bin/myapp:processRequest {
  printf("processRequest called from PID %d\n", pid);
}'

Cilium: eBPF for Kubernetes Networking

Cilium is a Kubernetes CNI that replaces iptables-based networking with eBPF. Beyond the performance benefits, it provides deep observability via Hubble.

Installing Cilium on K3s

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Install Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all \
  "https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz"
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin

# Install Cilium with Hubble enabled
cilium install \
  --set kubeProxyReplacement=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

# Verify installation
cilium status --wait

Hubble: Network Flow Visibility

Hubble gives you per-flow network visibility across all pods — who’s talking to whom, which connections are being dropped by policy, and latency between services.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# Install Hubble CLI
export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --fail --remote-name-all \
  "https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz"
sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin

# Port-forward to the Hubble relay
cilium hubble port-forward &

# Observe all flows
hubble observe

# Filter to a specific namespace
hubble observe --namespace production

# Filter to a specific pod
hubble observe --pod frontend

# Show only dropped flows (policy violations)
hubble observe --verdict DROPPED

# Show HTTP flows with status codes
hubble observe --protocol http

# Example output:
# Mar 26 10:23:41.123 [production/frontend] -> [production/backend] to-endpoint FORWARDED
# Mar 26 10:23:41.124 [production/backend] -> [production/database] to-endpoint FORWARDED
# Mar 26 10:23:42.001 [production/rogue-pod] -> [kube-system/etcd] to-endpoint DROPPED

Hubble UI

1
2
3

# Open Hubble's graphical service dependency map
cilium hubble ui
# Opens http://localhost:12000 — a real-time service map showing flows between pods

The Hubble UI shows a live service dependency graph: every service in your cluster, every connection, and whether flows are allowed or dropped. This alone is worth deploying Cilium — you’ll immediately see connections you didn’t know existed.

Network Policy with Cilium

Cilium extends Kubernetes NetworkPolicy with L7 (application-layer) awareness:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Allow frontend to reach backend only on GET /api/*
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: frontend-to-backend
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: /api/.*

With standard Kubernetes NetworkPolicy you can only filter by IP and port. Cilium’s L7 policies filter by HTTP method, path, headers — and all enforcement decisions are visible in Hubble.

Practical Investigation Examples

Why Is My Service Slow?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# 1. First — what are the latency distributions?
sudo biolatency-bpfcc -D   # disk I/O per device

# 2. Is it syscall overhead?
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_read,
tracepoint:syscalls:sys_enter_write
/comm == "myapp"/
{ @[probe] = count(); }' -c "sleep 10"

# 3. Profile CPU — where is time actually going?
sudo profile-bpfcc -F 99 -p $(pgrep myapp) 30 > stacks.txt
# Generate flame graph...

# 4. Is it network?
sudo tcpretrans-bpfcc   # TCP retransmissions — indicates network packet loss
sudo tcpconnlat-bpfcc   # connection establishment latency

What Is a Container Doing?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# Get the container's main PID
PID=$(docker inspect --format '{{.State.Pid}}' mycontainer)

# Trace all syscalls it makes
sudo bpftrace -e "tracepoint:raw_syscalls:sys_enter /pid == $PID/ {
  @[ksym(args->id)] = count();
}"

# What files is it opening?
sudo opensnoop-bpfcc -p $PID

# What processes is it spawning?
sudo execsnoop-bpfcc -p $PID

# What network connections is it making?
sudo tcpconnect-bpfcc -p $PID

Detecting a Security Incident

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# Watch for suspicious executions (base64, curl piped to sh, etc.)
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_execve {
  $cmd = str(args->filename);
  if ($cmd == "/usr/bin/base64" ||
      $cmd == "/bin/nc" ||
      $cmd == "/usr/bin/wget") {
    printf("ALERT: suspicious exec by PID %d (%s): %s\n",
           pid, comm, $cmd);
  }
}'

# Watch for privilege escalation attempts
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_setuid {
  if (args->uid == 0) {
    printf("ALERT: PID %d (%s) attempting setuid(0)\n", pid, comm);
  }
}'

# Watch for /etc/shadow reads
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_openat
/str(args->filename) == "/etc/shadow"/
{
  printf("ALERT: PID %d (%s) opened /etc/shadow\n", pid, comm);
}'

Parca: Always-On CPU Profiling

Parca uses eBPF to continuously profile every process on a host — no instrumentation, no restarts, no overhead worth measuring.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Run Parca server (stores profiles)
docker run -d \
  --name parca \
  -p 7070:7070 \
  -v /tmp/parca:/var/lib/parca \
  ghcr.io/parca-dev/parca:latest

# Run Parca agent on each node (collects profiles via eBPF)
docker run -d \
  --name parca-agent \
  --privileged \
  --pid host \
  -v /sys:/sys \
  -v /proc:/proc \
  ghcr.io/parca-dev/parca-agent:latest \
  --node=my-host \
  --store-address=http://parca:7070 \
  --insecure

Parca’s UI at http://localhost:7070 lets you query CPU profiles for any process over any time range. When you get a latency spike at 3 AM, you can look back and see exactly what code was running.

Key eBPF Concepts Summarized

Maps: Shared data structures between eBPF programs and user space. Types include hash maps, arrays, ring buffers, LRU maps, and per-CPU variants. Ring buffers (BPF_MAP_TYPE_RINGBUF) are the modern way to stream events to user space with minimal overhead.

BTF (BPF Type Format): Debug type information embedded in the kernel binary. Enables CO-RE — eBPF programs compiled on one machine can run on kernels with different struct layouts.

BPF CO-RE: Programs use bpf_core_read() macros instead of direct struct access. The loader applies relocations based on BTF, adjusting field offsets for the running kernel.

Verifier limits: eBPF programs have complexity limits — bounded loops only (since kernel 5.3, bounded loops are allowed), no unbounded recursion, limited stack depth (512 bytes). Very complex programs may be rejected.

Overhead: Well-written eBPF programs typically add < 1% CPU overhead. Poorly written ones (e.g., printing to user space in a hot path, complex maps in high-frequency hooks) can add more. Profile your eBPF programs the same way you’d profile any code.

eBPF has fundamentally changed what’s possible in production observability. Five years ago, getting a CPU flame graph required sampling with perf and hoping the stack unwinding worked. Getting per-flow network visibility required a service mesh. Tracking every file open by a container required strace with its prohibitive overhead.

Today, bpftrace one-liners answer questions that used to take hours, Cilium/Hubble gives you Kubernetes network visibility with no application changes, and tools like Parca profile your entire fleet continuously. The investment in learning eBPF pays off every time you open a bpftrace one-liner instead of staring at metrics trying to guess where the problem is.

Comments