Kubernetes Networking Internals: kube-proxy, iptables vs eBPF, CNI Plugins, and DNS

Kubernetes networking has a reputation for being mysterious. Services seem to magically route traffic to pods, DNS names resolve to cluster IPs that don’t exist on any interface, and yet somehow packets find their way. This guide tears open the black box: how the pod network model works, what kube-proxy actually does with iptables, how eBPF is replacing it, what CNI plugins do, and how CoreDNS resolves names inside the cluster.

Understanding this isn’t just academic. It’s the difference between guessing when a NetworkPolicy blocks unexpected traffic, and knowing exactly which iptables rule or eBPF map entry is responsible.

The Kubernetes Network Model

Kubernetes enforces a flat network model with three requirements:

1. All pods can communicate with all other pods without NAT
2. All nodes can communicate with all pods without NAT
3. The IP a pod sees for itself is the same IP other pods see for it

This sounds simple but has profound implications. There’s no inherent NAT between pods — if pod A has IP 10.244.1.5, any other pod reaching 10.244.1.5 reaches pod A directly. No port mapping, no masquerading.

This is different from Docker’s default networking where containers get private IPs behind a NAT bridge. Kubernetes requires that each pod gets a real, routable IP on a cluster-wide pod network.

The Three Distinct Networks

A Kubernetes cluster has three address spaces that never overlap:

Node Network:    192.168.1.0/24    (physical/VM network)
Pod Network:     10.244.0.0/16     (virtual, one /24 per node)
Service Network: 10.96.0.0/12     (virtual, only in iptables/eBPF maps)

Node network: real IP addresses on real network interfaces — how nodes communicate with each other.

Pod network: virtual IP addresses assigned to pods. Each node gets a subnet (e.g., node 1 gets 10.244.1.0/24, node 2 gets 10.244.2.0/24). Pods on the same node communicate via a local bridge. Pods on different nodes need a mechanism to route across nodes — this is what CNI plugins provide.

Service network: the most mysterious one. ClusterIPs live in this range but don’t exist on any network interface anywhere. They only exist as rules in iptables or entries in eBPF maps. When a pod sends a packet to a ClusterIP, the kernel intercepts it and rewrites the destination to a real pod IP before the packet leaves the host.

How Pods Get Network Interfaces

When kubelet creates a pod, it calls the configured CNI plugin to set up networking. The sequence:

1. kubelet creates the pod’s network namespace (/var/run/netns/<pod-id>)
2. kubelet calls the CNI plugin’s ADD command with the namespace path
3. CNI plugin creates a veth pair — one end inside the pod namespace, one end on the host
4. CNI plugin assigns the pod its IP address (from the node’s pod subnet)
5. CNI plugin sets up routes so traffic to other pods is forwarded correctly
6. CNI plugin installs any necessary iptables rules

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# See pod network namespaces on a node
ls /var/run/netns/

# Inspect a pod's network namespace
POD_NS=$(crictl inspect <container-id> | jq -r '.info.runtimeSpec.linux.namespaces[] | select(.type=="network") | .path')
ip netns exec $POD_NS ip addr

# See veth pairs on the host
ip link show type veth

# See the bridge connecting pod veths (with flannel/bridge CNI)
bridge link show
ip addr show cni0   # The bridge interface

Inside the pod namespace:

eth0: 10.244.1.5/24   ← pod's veth end
      default via 10.244.1.1   ← gateway is the bridge on the host

On the host:

cni0: 10.244.1.1/24   ← bridge (gateway for all pods on this node)
veth3a2b1c: ...       ← host end of the veth pair for this pod

CNI Plugins: How Pods on Different Nodes Communicate

The hardest problem in the pod network: pod on node 1 (10.244.1.5) sending to pod on node 2 (10.244.2.7). Node 1’s kernel needs to know how to reach 10.244.2.0/24. CNI plugins solve this in three main ways.

Overlay Networks (Flannel VXLAN, Calico VXLAN)

An overlay wraps the pod IP packet in an outer packet. The inner packet has pod IPs; the outer packet uses node IPs that the underlying network understands.

Inner packet: src=10.244.1.5  dst=10.244.2.7
Outer packet: src=192.168.1.10 dst=192.168.1.11  (node IPs)
              VXLAN header (UDP port 8472)

Node 1’s VXLAN device (flannel.1) intercepts packets destined for 10.244.2.0/24, encapsulates them in UDP/VXLAN, and sends to node 2. Node 2’s VXLAN device decapsulates and forwards to the pod bridge.

Pros: works on any network — the overlay handles routing regardless of the underlay topology. Cons: encapsulation overhead (50+ bytes per packet), slightly higher CPU usage for encap/decap.

1
2
3
4

# Flannel VXLAN setup
ip link show flannel.1          # The VXLAN device
ip route show                   # Routes via flannel.1 for remote pod subnets
bridge fdb show dev flannel.1   # VTEP (VXLAN tunnel endpoint) MAC table

Native Routing (Calico BGP, Cilium native)

Routes are distributed directly — no encapsulation. Each node advertises its pod subnet via BGP (Calico) or by programming the kernel routing table directly (Cilium with native routing).

Node 1 routing table:
  10.244.1.0/24 dev cni0          # Local pods — via bridge
  10.244.2.0/24 via 192.168.1.11  # Remote node 2's pods — direct route
  10.244.3.0/24 via 192.168.1.12  # Remote node 3's pods — direct route

When a pod sends to 10.244.2.7, the kernel routes directly to node 2 without any encapsulation. The underlying network must be able to route between pod subnets — typically requiring the nodes to be on the same L2 segment, or a cloud VPC with routes programmed.

Pros: no overhead, full MTU for pods, simpler packet path. Cons: requires underlay network cooperation (not always possible).

1
2
3

# Calico BGP routing
calicoctl node status             # BGP peer status
ip route show proto bird          # Routes learned via BGP

WireGuard Encryption (Calico, Cilium)

Some CNI plugins can encrypt pod-to-pod traffic using WireGuard at the CNI level — transparent to applications.

1
2
3
4
5

# Cilium WireGuard status
cilium encrypt status
# WireGuard: enabled
# Keys in use: 1
# Peers: 3

CNI Plugin Comparison

For new clusters: Cilium is generally the best choice. For existing clusters with simple needs: Calico is battle-tested and widely supported.

kube-proxy: Making Services Work

Services are Kubernetes’ load balancing abstraction. A Service gets a stable ClusterIP that doesn’t change even as pods come and go. kube-proxy is responsible for making that ClusterIP work.

kube-proxy watches the Kubernetes API for Service and Endpoints (or EndpointSlice) objects. When a Service is created or its endpoints change, kube-proxy updates the local data plane on every node.

iptables Mode (Default)

In iptables mode, kube-proxy translates Services into iptables NAT rules. When a pod sends to a ClusterIP, iptables intercepts the packet in the PREROUTING chain, performs DNAT (Destination NAT) to a real pod IP, and forwards it.

Let’s trace a ClusterIP Service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# Service: my-service, ClusterIP: 10.96.100.50, Port: 80
# Backends: 10.244.1.5:8080, 10.244.2.7:8080

# kube-proxy creates these rules:

# In PREROUTING / OUTPUT — send to KUBE-SERVICES chain
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT     -m comment --comment "kubernetes service portals" -j KUBE-SERVICES

# Match the ClusterIP, jump to the service chain
-A KUBE-SERVICES -d 10.96.100.50/32 -p tcp --dport 80 \
  -m comment --comment "default/my-service cluster IP" \
  -j KUBE-SVC-ABCD1234

# Load balance with random selection (statistic module)
# 50% chance to backend 1
-A KUBE-SVC-ABCD1234 \
  -m statistic --mode random --probability 0.5 \
  -j KUBE-SEP-XXXX1111

# Otherwise (remaining 100%) to backend 2
-A KUBE-SVC-ABCD1234 \
  -j KUBE-SEP-XXXX2222

# Backend 1: DNAT to 10.244.1.5:8080
-A KUBE-SEP-XXXX1111 \
  -p tcp -m tcp \
  -j DNAT --to-destination 10.244.1.5:8080

# Backend 2: DNAT to 10.244.2.7:8080
-A KUBE-SEP-XXXX2222 \
  -p tcp -m tcp \
  -j DNAT --to-destination 10.244.2.7:8080

1
2
3
4
5
6
7
8

# Inspect on a real node
iptables -t nat -L KUBE-SERVICES -n --line-numbers
iptables -t nat -L KUBE-SVC-<hash> -n -v
iptables -t nat -L KUBE-SEP-<hash> -n -v

# Count all kube-proxy rules
iptables -t nat -L | grep -c KUBE
# Often thousands of rules on large clusters

The iptables Scaling Problem

iptables rules are evaluated sequentially in kernel space. With 10,000 Services and 3 pods each, that’s ~30,000 NAT rules. Every new packet traverses the entire chain until a match is found. At thousands of Services, this becomes measurable latency and CPU overhead. Adding or removing a single rule requires locking and rewriting the entire ruleset.

Benchmarks show iptables rule processing time growing roughly linearly — at 10,000 Services, rule traversal can add 10-20µs per packet.

IPVS Mode

IPVS (IP Virtual Server) is a kernel load balancer designed exactly for this use case. Instead of chains of rules, it uses hash tables — O(1) lookup regardless of the number of Services.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Enable IPVS mode in kube-proxy config
# (set via kube-proxy ConfigMap or --proxy-mode=ipvs flag)

# Inspect IPVS rules
ipvsadm -ln

# Output:
# IP Virtual Server version 1.2.1 (size=4096)
# Prot LocalAddress:Port Scheduler Flags
#   -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
# TCP  10.96.100.50:80 rr
#   -> 10.244.1.5:8080             Masq    1      0          0
#   -> 10.244.2.7:8080             Masq    1      0          0

IPVS supports multiple load balancing algorithms: round-robin (rr), least connections (lc), source hashing (sh), and more. kube-proxy in IPVS mode uses rr by default.

Caveat: IPVS still uses iptables for some operations (masquerade, NodePort). It’s not a complete replacement but significantly better at scale.

kube-proxy Replacement with Cilium eBPF

Cilium can replace kube-proxy entirely using eBPF maps instead of either iptables or IPVS. eBPF maps are hash tables in kernel memory — O(1) lookup, lock-free updates, no full-table rewrites.

1
2
3
4
5
6
7
8
9

# Cilium kube-proxy replacement status
cilium status | grep KubeProxyReplacement
# KubeProxyReplacement: True

# Inspect eBPF service maps
cilium service list

# See the eBPF programs attached to interfaces
bpftool prog list | grep cilium

The eBPF approach handles Service load balancing at the socket layer (SOCK_OPS hook) — before the packet is even formed. This means connection-level load balancing happens at the syscall boundary, with zero packet-level overhead.

1
2
3
4

# Cilium socket-level load balancing
# Packets are load balanced at connect() time, not per-packet
bpftool prog list | grep sock
# cgroup_skb  id=100  name cil_sock4_connect

NodePort and LoadBalancer Services

NodePort

A NodePort Service opens the same port on every node. Traffic arriving on any node’s IP at that port is forwarded to the Service’s pods.

1
2
3
4
5
6
7
8
9

# NodePort rules in iptables
iptables -t nat -L KUBE-NODEPORTS -n

# Traffic arrives on node at port 30080
# → KUBE-SVC-... chain → KUBE-SEP-... → pod

# With Cilium, NodePort is handled by XDP on the NIC
# (hardware-level, before the packet enters the kernel network stack)
cilium status | grep NodePort

LoadBalancer

For LoadBalancer type Services, kube-proxy (or Cilium) doesn’t provision the actual load balancer — that’s done by a cloud controller manager or MetalLB for bare-metal. Once the external IP is assigned, traffic flows: External IP → Node → NodePort → Pod.

1
2
3

# MetalLB for bare-metal LoadBalancer services
kubectl get ipaddresspools -n metallb-system
kubectl get l2advertisements -n metallb-system

DNS Resolution in Kubernetes

Every pod in Kubernetes gets /etc/resolv.conf configured to point at CoreDNS:

# /etc/resolv.conf inside a pod
nameserver 10.96.0.10          # CoreDNS ClusterIP
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The Search Domain Chain

ndots:5 means: if a name has fewer than 5 dots, try appending the search domains before attempting an absolute lookup. For a lookup of my-service from a pod in the default namespace:

1. my-service.default.svc.cluster.local  → resolves ✓ (stop here)
2. my-service.svc.cluster.local          → would try next if (1) failed
3. my-service.cluster.local              → would try next
4. my-service.                           → would try last (absolute)

This is why curl http://my-service works from within the same namespace — the search domain appends .default.svc.cluster.local and CoreDNS resolves it.

For cross-namespace: curl http://my-service.other-namespace also works because the search domain appends .svc.cluster.local:

• my-service.other-namespace.default.svc.cluster.local → fails
• my-service.other-namespace.svc.cluster.local → resolves ✓

The fully qualified form always works and avoids the search domain chain:

curl http://my-service.other-namespace.svc.cluster.local

ndots:5 Performance Problem

ndots:5 causes excessive DNS lookups for external names. api.stripe.com has 2 dots → triggers 4 failed lookups before the final correct one:

api.stripe.com.default.svc.cluster.local   → NXDOMAIN
api.stripe.com.svc.cluster.local           → NXDOMAIN
api.stripe.com.cluster.local               → NXDOMAIN
api.stripe.com.                            → resolves ✓

Fix with fully qualified names in application configs (trailing dot) or override per-pod:

1
2
3
4
5

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"    # Only append search domains if fewer than 2 dots

CoreDNS Internals

CoreDNS is a plugin-based DNS server. Its configuration (Corefile) is a ConfigMap in kube-system:

1

kubectl -n kube-system get configmap coredns -o yaml

Default Corefile:

.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

Key plugins:

• kubernetes: handles *.cluster.local and reverse DNS for pod/service IPs. Talks to the Kubernetes API to get Service/Endpoint records.
• forward: forwards non-cluster queries to the upstream resolver (node’s /etc/resolv.conf)
• cache: caches responses for up to 30 seconds (TTL)
• loadbalance: randomizes the order of A/AAAA records in responses

DNS Record Types

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# From inside a pod:

# Service → ClusterIP (A record)
nslookup my-service.default.svc.cluster.local
# Address: 10.96.100.50

# Headless service → pod IPs directly (A records, no ClusterIP)
nslookup my-headless-service.default.svc.cluster.local
# Address: 10.244.1.5
# Address: 10.244.2.7
# Address: 10.244.3.9

# Pod DNS (if pods: insecure in Corefile)
# pod-ip with dashes . namespace .pod. cluster.local
nslookup 10-244-1-5.default.pod.cluster.local
# Address: 10.244.1.5

# SRV records for named ports
nslookup _http._tcp.my-service.default.svc.cluster.local
# service = 0 100 8080 my-service.default.svc.cluster.local

Custom DNS with Stub Zones

Override DNS for specific domains — useful for on-premises split DNS or service discovery outside the cluster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# ConfigMap patch for custom stub zones
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
        }
        # Forward internal.company.com to on-prem DNS
        forward internal.company.com 10.0.0.53 10.0.0.54
        # Forward everything else to public resolvers
        forward . 8.8.8.8 8.8.4.4
        cache 30
        loop
        reload
        loadbalance
    }

1
2
3

# Apply and reload (CoreDNS watches the ConfigMap)
kubectl -n kube-system apply -f coredns-configmap.yaml
kubectl -n kube-system rollout restart deployment/coredns

Debugging Kubernetes Networking

Pod-to-Pod Connectivity

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Test from pod A to pod B directly (bypassing Service)
kubectl exec -it pod-a -- curl http://10.244.2.7:8080/health

# If that fails but the pod is healthy, check CNI
# Look at routes in the source pod
kubectl exec -it pod-a -- ip route

# Check routes on the source node
NODE=$(kubectl get pod pod-a -o jsonpath='{.spec.nodeName}')
kubectl debug node/$NODE -it --image=nicolaka/netshoot -- ip route

# Check if overlay is working
kubectl debug node/$NODE -it --image=nicolaka/netshoot -- \
  ping -c 3 10.244.2.1   # Gateway of destination node's pod subnet

Service Connectivity

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Test ClusterIP directly
kubectl exec -it pod-a -- curl http://10.96.100.50:80

# If ClusterIP fails, test endpoint directly
kubectl get endpoints my-service
kubectl exec -it pod-a -- curl http://10.244.1.5:8080

# Check kube-proxy is running and synced
kubectl -n kube-system get pods -l k8s-app=kube-proxy
kubectl -n kube-system logs -l k8s-app=kube-proxy | tail -20

# Check iptables rules exist for the Service
iptables -t nat -L KUBE-SERVICES -n | grep 10.96.100.50

DNS Debugging

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# Run a debug pod with DNS tools
kubectl run dns-test --image=nicolaka/netshoot --rm -it -- bash

# Test DNS resolution
nslookup my-service
nslookup my-service.default.svc.cluster.local
nslookup kubernetes.default.svc.cluster.local   # Should always work

# Check which server is being used
cat /etc/resolv.conf

# Test CoreDNS directly
dig @10.96.0.10 my-service.default.svc.cluster.local

# Check CoreDNS logs
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50

# Enable CoreDNS query logging temporarily (careful — very verbose)
kubectl -n kube-system edit configmap coredns
# Add 'log' plugin to the Corefile

Packet Capture Inside a Pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Use netshoot for tcpdump inside a pod's network namespace
kubectl debug -it my-pod --image=nicolaka/netshoot --target=my-container -- \
  tcpdump -i eth0 -n port 8080

# Or exec into an existing debug container
kubectl exec -it my-pod -c netshoot -- tcpdump -i any -n

# Capture on the host side (veth)
# Find the veth for the pod
POD_IP=$(kubectl get pod my-pod -o jsonpath='{.status.podIP}')
# On the node:
ip route get $POD_IP   # Shows which veth interface
tcpdump -i veth3a2b1c -n

NetworkPolicy Troubleshooting

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# With Cilium — the best tooling
hubble observe --pod default/my-pod --verdict DROPPED --follow

# Simulate a policy decision
kubectl -n kube-system exec -it ds/cilium -- \
  cilium policy trace \
    --src-k8s-pod default/client-pod \
    --dst-k8s-pod default/server-pod \
    --dport 8080/TCP

# With Calico
calicoctl get networkpolicy -o wide
kubectl -n kube-system exec -it ds/calico-node -- \
  calico-node -felix-diag-dump 2>/dev/null | grep -A 5 "Policy"

eBPF vs iptables: The Future of Kubernetes Networking

The industry is clearly moving toward eBPF-based networking for Kubernetes. The advantages compound:

At 1,000+ Services or 10,000+ pods, the iptables performance cliff becomes measurable. At 10,000+ Services, it’s a serious problem. eBPF-based solutions scale linearly or better.

The migration path: you don’t have to start fresh. Cilium can be installed in existing clusters in migration mode, replacing kube-proxy incrementally. Run cilium install --set kubeProxyReplacement=true on a new cluster, or follow Cilium’s kube-proxy migration guide for existing clusters.

Quick Reference

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# Pod networking
ip addr show                        # Pod's IP inside its namespace
ip route show                       # Pod's routing table
bridge link show                    # veth pairs on node

# Services / kube-proxy
iptables -t nat -L KUBE-SERVICES -n  # All Service VIPs
iptables -t nat -L KUBE-SVC-<hash>   # Per-service load balance rules
ipvsadm -ln                          # IPVS rules (if IPVS mode)
cilium service list                  # eBPF service map (Cilium)

# DNS
cat /etc/resolv.conf                 # Pod's DNS config
nslookup <service>                   # Resolve a Service
dig @10.96.0.10 <name>               # Query CoreDNS directly
kubectl -n kube-system logs -l k8s-app=kube-dns  # CoreDNS logs

# CNI
ls /etc/cni/net.d/                   # CNI config files
cat /etc/cni/net.d/10-flannel.conflist
ls /opt/cni/bin/                     # CNI plugin binaries

# Debugging
kubectl run netshoot --image=nicolaka/netshoot --rm -it -- bash
kubectl debug node/<name> -it --image=nicolaka/netshoot
hubble observe --verdict DROPPED --follow  # Cilium drop events

Kubernetes networking is built from standard Linux primitives — namespaces, veth pairs, bridges, iptables, and increasingly eBPF. Once you see the layers clearly, the “magic” disappears and you’re left with familiar tools you already know how to debug.