LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

eBPF for Networking

14 min read Networking Deep Dives
ebpfxdpnetworkinglinux-kernelload-balancingtc-bpf
Contents

For thirty years the Linux network stack was a fixed pipeline: a packet arrived at the NIC, the driver built an sk_buff, and the packet marched through netfilter hooks, routing, and the socket layer in an order you could configure with iptables rules but could not fundamentally change. eBPF broke that. It let you attach verified, JIT-compiled programs at specific points in the packet’s journey and have them make a verdict — pass, drop, redirect, rewrite — at native speed, before the kernel has spent the cycles to do its normal work. The headline result is that you can drop a DDoS flood at 20+ million packets per second on a single commodity box, load-balance Layer 4 traffic without a kernel module, and replace a tangle of iptables rules with a hash-map lookup. The catch, which most eBPF marketing skips, is that where you attach the program decides almost everything about what it can do and what it costs, and the two main attach points — XDP and tc — are not interchangeable. This post is about that datapath: where the hooks sit, what each can see, and what you actually pay to run code there.

This is deliberately the networking datapath angle. eBPF’s tracing and observability story — kprobes, bpftrace, BCC, profiling — is a separate world covered in the eBPF for observability post, and Cilium’s full role as a Kubernetes CNI is covered in Cilium as a full CNI. Here the question is narrower and more physical: when a packet hits your NIC, what can you make the kernel do to it, and how early?

What eBPF is, in one section

eBPF is a small register-based virtual machine inside the kernel. You write a program (usually in a restricted C, compiled to BPF bytecode with clang/LLVM), and before the kernel will run it, a verifier statically proves that it terminates (bounded loops only), never dereferences an unchecked pointer, and only touches memory it is allowed to. If the proof passes, a JIT compiler turns the bytecode into native machine code, so the program runs at roughly the speed of compiled C — there is no interpreter tax in the hot path. Programs are stateless between invocations except for maps: typed key/value structures (hash maps, arrays, LRU maps, per-CPU maps, ring buffers) that persist in the kernel and are shared with user space. A load-balancer’s backend table, a DDoS blocklist, a per-flow counter — all of these live in maps that a user-space control plane writes and the datapath program reads on every packet.

The verifier is the entire reason eBPF is safe enough to put in the packet path. A bad kernel module panics the box; a bad eBPF program is rejected at load time with a (famously cryptic) verifier error. The cost is that the verifier is conservative — it rejects programs it cannot prove safe even when they are — which is the source of most of the learning curve. You will spend real time rephrasing correct code until the verifier is convinced, and the error messages are improving but still rough.

The hook ladder: where you can attach, and why it matters

A packet entering a Linux box passes a fixed sequence of points, and eBPF can attach at several of them. The order is the single most important thing to internalize, because it determines what each program can see and how much work the kernel has already done — and therefore wasted — by the time your code runs.

INGRESS PATH OF A RECEIVED PACKET (earliest to latest)

  NIC hardware
    | (DMA into ring buffer)
  [ XDP ]  <-- runs on the raw DMA'd frame, BEFORE sk_buff allocation
    |          verdicts: XDP_DROP / XDP_PASS / XDP_TX / XDP_REDIRECT
    | (kernel allocates sk_buff -- the expensive part)
  [ tc ingress / clsact ]  <-- runs on a fully-built sk_buff
    |          verdicts: TC_ACT_OK / SHOT / REDIRECT, can read/write metadata
  netfilter (conntrack, nftables)
    |
  routing decision
    |
  [ socket / cgroup hooks ]  <-- per-socket, sees the connection
    |
  application

EGRESS:  socket -> routing -> [ tc egress / clsact ] -> driver -> NIC
         (XDP is ingress-only; there is no XDP egress hook)

The crucial line is “before sk_buff allocation.” The sk_buff is the kernel’s heavyweight per-packet metadata structure, and allocating and initializing one is a large fraction of the cost of receiving a packet. XDP runs before that allocation exists — it works directly on the page the NIC DMA’d the frame into. That is why XDP can drop packets so cheaply: a dropped packet never gets an sk_buff, never touches conntrack, never costs anything beyond the few instructions of your program. It is also why XDP is limited: there is no sk_buff, so you have no access to conntrack state, no socket, no routing result — just raw bytes and a couple of metadata fields. You are parsing Ethernet/IP/TCP headers by hand, with bounds checks the verifier forces on every access.

tc (traffic control) eBPF, attached via the clsact qdisc, runs one step later, after the sk_buff is built. You pay the allocation cost, but in exchange you get a fully-formed packet with metadata, you can attach on both ingress and egress (XDP is ingress-only), and you can mangle and redirect with the full kernel context available. tc-bpf is where traffic shaping, egress policy, and anything needing connection context lives.

Property XDP tc (clsact) eBPF
Position Driver, pre-sk_buff Post-sk_buff, in stack
Direction Ingress only Ingress and egress
Sees Raw frame bytes only Full sk_buff + metadata
Drop cost Lowest possible Higher (alloc already paid)
Can shape/queue No Yes (with qdiscs)
Typical job DDoS drop, L4 LB, fast redirect Shaping, egress policy, NAT-ish rewrite
Peak drop rate ~tens of Mpps/core Lower (full stack entered)

XDP: the driver-edge fast path

XDP (eXpress Data Path) is the reason “eBPF networking” gets benchmarked at line rate. Your program runs once per received frame, returns one of four verdicts, and the kernel obeys it immediately:

  • XDP_DROP — free the frame, do nothing else. This is the DDoS primitive: a flood of malicious packets is discarded for the cost of a header parse and a map lookup, never entering the stack.
  • XDP_PASS — let the frame continue up to the normal stack (gets an sk_buff and proceeds).
  • XDP_TX — bounce the frame back out the same NIC, optionally after rewriting it. This is how you build a stateless reflector or a one-armed load balancer.
  • XDP_REDIRECT — send the frame to another NIC, a CPU queue, or an AF_XDP user-space socket. This is the basis of fast forwarding and userspace packet processing.

XDP comes in three modes, and which one you get depends on driver support. Native XDP runs in the driver’s receive path and is the fast one — it requires the driver to implement the XDP hook (most modern 10G+ NICs do: mlx5, i40e, ice, ena, virtio-net). Offloaded XDP pushes the program onto a SmartNIC’s own processor (Netronome was the classic example), so the host CPU never touches the dropped packets at all — the rarest and fastest mode. Generic (SKB) XDP is a fallback the kernel provides for any driver by running the program after sk_buff allocation; it works everywhere but throws away XDP’s whole performance advantage, so it is for development and testing, not production.

A minimal XDP program that drops all UDP traffic to port 53 shows the shape of the thing — note the relentless bounds checks the verifier demands before every header access:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

SEC("xdp")
int xdp_drop_dns(struct xdp_md *ctx)
{
    void *data     = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;

    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)        /* verifier: prove in-bounds */
        return XDP_PASS;
    if (eth->h_proto != bpf_htons(ETH_P_IP))
        return XDP_PASS;

    struct iphdr *ip = (void *)(eth + 1);
    if ((void *)(ip + 1) > data_end)
        return XDP_PASS;
    if (ip->protocol != IPPROTO_UDP)
        return XDP_PASS;

    struct udphdr *udp = (void *)ip + ip->ihl * 4;
    if ((void *)(udp + 1) > data_end)
        return XDP_PASS;

    if (udp->dest == bpf_htons(53))
        return XDP_DROP;                      /* gone, no sk_buff ever built */
    return XDP_PASS;
}

Load it and watch it work with the standard tooling:

1
2
3
4
5
6
7

# attach in native mode (falls back to generic only if forced)
ip link set dev eth0 xdpdrv obj xdp_drop_dns.o sec xdp
# or with bpftool, and inspect what is loaded
bpftool prog show
bpftool net show dev eth0
# detach
ip link set dev eth0 xdpdrv off

The honest performance picture: native XDP on a single modern core drops on the order of 15-25 million packets per second, scaling with cores up to NIC and PCIe limits — numbers a userspace firewall cannot approach because each dropped packet there costs a full sk_buff and stack traversal. This is why Cloudflare, Facebook/Meta, and the major clouds run XDP at the edge for volumetric DDoS scrubbing: it is the cheapest place in the entire system to say no to a packet.

tc-bpf: shaping, egress, and the rest of the datapath

XDP cannot shape traffic — it has no queues, runs ingress-only, and works before the kernel has a packet structure to enqueue. Everything involving rate limiting, prioritization, egress policy, or connection-aware rewriting lives in tc-bpf, attached to the clsact qdisc, which provides clean ingress and egress hooks without the historical baggage of classful qdiscs.

1
2
3
4
5

# create the clsact qdisc, then attach an eBPF classifier on egress
tc qdisc add dev eth0 clsact
tc filter add dev eth0 egress bpf da obj shaper.o sec tc
# inspect
tc filter show dev eth0 egress

A tc program returns TC_ACT_OK (continue), TC_ACT_SHOT (drop), TC_ACT_REDIRECT (send elsewhere), and friends, and because it holds a real sk_buff it can read and write the packet’s metadata, mark it (skb->mark), adjust headers, and cooperate with the kernel’s queueing disciplines for actual rate control. The genuinely modern way to shape egress, though, is not classful HTB but EDT (Earliest Departure Time): a tc-bpf program computes a timestamp for when each packet should leave, stamps it into skb->tstamp, and the fq (fair queue) qdisc releases packets at those times. This pushes the rate-limiting policy into a flexible eBPF program while leaving the mechanism (the timing wheel) in the kernel, and it is how Cilium implements bandwidth limits per pod. It scales far better than walking a tree of HTB classes under load.

The division of labor is clean once you see it: XDP is your ingress bouncer — fast, dumb, ingress-only, drops and redirects. tc-bpf is your in-stack traffic engineer — shaping, egress policy, marking, connection-aware rewriting. Real systems use both: XDP to shed obvious garbage at the door, tc to manage the traffic that gets in.

What gets built on this: load balancing and DDoS

The two flagship production uses of eBPF networking both fall out of XDP directly.

L4 load balancing. Meta’s Katran and Cilium’s built-in load balancer are XDP programs. A virtual IP (VIP) is announced; packets to the VIP hit an XDP program that hashes the 5-tuple, looks the result up against a consistent-hashing backend table stored in a BPF map, encapsulates or rewrites the packet, and XDP_TX/XDP_REDIRECTs it to a real server — all before an sk_buff exists. Because the backend table is a map, the control plane adds and drains servers by updating map entries with zero datapath downtime. Consistent hashing (Maglev-style) keeps existing flows pinned to the same backend even as the backend set changes, which is what makes it usable for stateful TCP. This replaces a rack of dedicated L4 load-balancer appliances with software on the same servers doing the work, at line rate.

DDoS mitigation. The XDP drop primitive is the whole story: a control plane watches traffic, identifies attack signatures (bad source prefixes, malformed packets, volumetric floods to a port), writes them into a blocklist map, and the XDP program drops matching packets for nearly free while passing legitimate traffic to the stack. Because the drop happens before sk_buff allocation and before conntrack, the attack cannot exhaust those resources — which is precisely the resource-exhaustion that brings down iptables-based defenses under a large flood.

XDP L4 LOAD BALANCER (stateless, map-driven)

  client pkt -> VIP
       |
   [ XDP prog on edge node ]
       |  hash(5-tuple) -> consistent-hash ring (BPF map)
       |  pick backend, encap/rewrite
       v
   XDP_REDIRECT --> real server (DSR: server replies direct to client)

  control plane writes backend map; datapath never blocks on it

Cilium stitches all of this together for Kubernetes — kube-proxy replacement via XDP/tc, network policy enforcement, bandwidth EDT shaping — and Tetragon adds security observability on top, but those are the products; the primitives above are what they are built from.

The honest operational reality

eBPF networking is not free, and the costs are specific:

  • Kernel version is a hard dependency. XDP basics need 4.8+, but the features you actually want — BPF_PROG_TYPE improvements, CO-RE (Compile Once, Run Everywhere) via BTF, EDT shaping, the better verifier — land across 5.x and keep improving. CO-RE and BTF (kernel type information embedded in the kernel, /sys/kernel/btf/vmlinux) are what let one compiled program run across kernel versions without recompiling against each kernel’s headers; without BTF you are back to brittle per-kernel builds. In practice you want a recent LTS kernel (5.15+, ideally 6.x) before committing.
  • Driver support gates native XDP. If your NIC driver does not implement the native XDP hook, you fall back to generic mode and lose the entire performance argument. Check before you architect around XDP — this bites people on older or exotic NICs and in some virtualized environments.
  • The verifier is the learning curve. Correct programs get rejected. Loops must be bounded, every pointer access needs a proven bounds check, and stack space is tight (512 bytes). The errors are improving but you will fight them, and complex programs require structuring code specifically to keep the verifier happy.
  • Debugging is harder than userspace. Your code runs in the kernel datapath. bpf_printk to the trace pipe, bpftool prog/map dumps, and xdpdump are the tools; there is no gdb on a running XDP program. A logic bug does not crash — it silently misroutes or drops packets, which is its own kind of painful.
  • XDP_TX/REDIRECT bypasses the stack you may rely on. A packet you XDP_TX never sees conntrack, never gets logged by your normal tooling, never hits tcpdump in the usual place. The performance comes precisely from skipping that machinery, so anything you needed from it you now have to provide yourself.

None of this argues against eBPF networking — at the scale where you need 20 Mpps DDoS drop or software L4 load balancing, there is no real alternative, and the major infrastructure operators run it because the economics are overwhelming. It argues for being clear-eyed: you are putting verified code in the kernel’s packet path, gaining enormous speed and flexibility, and paying for it in kernel-version discipline, driver dependency, verifier friction, and a harder debugging story. For a Kubernetes cluster that just wants a fast CNI and policy, lean on Cilium rather than writing XDP by hand. For a global edge that scrubs attacks and load-balances at line rate, the hand-written XDP datapath is the entire point — and the same hooks underpin both the observability and security stories that get layered on later.

Verdict

eBPF turned the Linux network stack from a fixed pipeline into a programmable one, and the whole engineering reality reduces to where you attach. XDP runs at the driver edge, before the kernel builds an sk_buff, which is why it drops DDoS floods and load-balances L4 traffic at tens of millions of packets per second per core — and also why it is blind to conntrack, sockets, and routing, working on raw bytes with a verifier checking every access. tc-bpf runs one step later on a full sk_buff, on both ingress and egress, which is where shaping (modern EDT-style), egress policy, and connection-aware rewriting belong. Build with both: XDP as the cheap bouncer at the door, tc as the traffic engineer inside. The costs are real and specific — a recent kernel, native-XDP driver support, the verifier’s learning curve, and kernel-datapath debugging — but at the scale where you need them, nothing else competes. If you run Kubernetes, get this through Cilium; if you run an edge, the hand-written XDP datapath is exactly the tool the big operators reach for, and now you know which hook does what.

Sources

Comments