Network Observability with Cilium and Hubble: Complete Visibility Into Your Kubernetes Network

March 26, 2026 18 min read Networking Deep Dives

kubernetesciliumhubbleobservabilitynetworkingebpf

Contents

One of the hardest problems in Kubernetes networking is answering a simple question: why can’t pod A talk to pod B? Traditional network debugging means guessing at iptables rules, running tcpdump inside containers, and piecing together what happened after the fact. Cilium and Hubble take a fundamentally different approach — using eBPF to observe every network flow at the kernel level, with identity-aware visibility that understands Kubernetes labels, namespaces, and DNS names rather than just IP addresses.

This guide walks through deploying Cilium as your CNI, enabling Hubble for observability, debugging real network policy problems, and building a complete visibility layer with metrics and dashboards.

Why Cilium?

Most Kubernetes CNI plugins implement networking by translating NetworkPolicy objects into iptables rules. This works, but has serious limitations:

Scale: iptables rules are O(n) — every new rule means re-scanning the entire chain. At thousands of pods, this becomes measurable overhead.
Observability: iptables drop packets silently. There’s no built-in way to ask “what got dropped and why?”
Identity: iptables works with IP addresses. In Kubernetes, pod IPs are ephemeral — they change on restart. Policies based on IP addresses are fragile.
Debugging: no visibility into what rules matched or why a connection failed.

Cilium replaces iptables with eBPF programs loaded directly into the kernel. Every packet decision happens in an eBPF program that understands Kubernetes identities (pod labels, service accounts, namespaces) natively. The same eBPF hooks that make policy decisions also record flow data — which is how Hubble works.

Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                    Kubernetes Node                       │
│                                                         │
│  Pod A ──────► eBPF hook (TC/XDP) ──────► Pod B        │
│                     │                                   │
│                     ▼                                   │
│              Flow events (perf ring buffer)             │
│                     │                                   │
│                     ▼                                   │
│              Hubble observer (per-node)                 │
│                     │                                   │
└─────────────────────┼───────────────────────────────────┘
                      │ gRPC
                      ▼
              Hubble Relay (cluster-wide aggregator)
                      │
              ┌───────┴──────────┐
              ▼                  ▼
        Hubble UI          Hubble CLI
        (browser)         (hubble observe)
              │
              ▼
        Prometheus + Grafana

Cilium agent: runs as a DaemonSet on every node. Manages eBPF programs, enforces network policy, provides the local Hubble observer.

Hubble: the observability layer built into Cilium. Two components:

Hubble observer: per-node gRPC server exposing flow data from the local eBPF ring buffer
Hubble Relay: aggregates flow data from all nodes into a single cluster-wide endpoint

Hubble UI: a web interface that renders real-time service dependency maps and flow tables.

Installation

Prerequisites

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Helm 3.x
helm version

# kubectl configured against your cluster
kubectl cluster-info

# Install cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin

Install Cilium with Hubble Enabled

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# Add the Helm repo
helm repo add cilium https://helm.cilium.io/
helm repo update

# Install Cilium with Hubble relay and UI
helm install cilium cilium/cilium \
  --version 1.16.0 \
  --namespace kube-system \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.metrics.enableOpenMetrics=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}" \
  --set prometheus.enabled=true \
  --set operator.prometheus.enabled=true

# Wait for Cilium to be ready
cilium status --wait

For existing clusters replacing an existing CNI, you need to restart all nodes or use Cilium’s migration guide. For new clusters (e.g., with kubeadm), skip the default CNI installation and install Cilium immediately after kubeadm init:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# kubeadm init without CNI
kubeadm init --skip-phases=addon/kube-proxy   # Cilium can replace kube-proxy too

# Then install Cilium with kube-proxy replacement
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=<API_SERVER_IP> \
  --set k8sServicePort=6443 \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

Verify Installation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# Check all Cilium pods are running
kubectl -n kube-system get pods -l k8s-app=cilium

# Run connectivity test (deploys test pods and validates networking)
cilium connectivity test

# Check Hubble status
cilium hubble ui &   # Opens the UI in your browser

# Install the hubble CLI
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz
sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin

# Port-forward Hubble relay and observe flows
cilium hubble port-forward &
hubble observe

The hubble CLI: Your Primary Debug Tool

hubble observe is the kubectl logs of networking — the first command you reach for when something is broken.

Basic Flow Observation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


# Stream all flows in real time
hubble observe --follow

# Last 100 flows
hubble observe --last 100

# Pretty-print with full details
hubble observe --last 50 --output json | jq .

# Filter by namespace
hubble observe --namespace production --follow

# Filter by pod name (supports regex)
hubble observe --pod frontend --follow
hubble observe --from-pod frontend --to-pod backend --follow

# Filter by verdict
hubble observe --verdict DROPPED --follow
hubble observe --verdict FORWARDED --follow

# Filter by protocol
hubble observe --protocol tcp --follow
hubble observe --protocol dns --follow

# Filter by port
hubble observe --port 5432 --follow   # All flows to/from port 5432

# Filter by IP
hubble observe --ip 10.0.1.42 --follow

# Filter by label selector
hubble observe --label app=frontend --follow

DNS Visibility

One of Hubble’s most useful features — see exactly what DNS queries pods are making and what they resolve to:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# All DNS flows
hubble observe --protocol dns --follow

# DNS for a specific pod
hubble observe --from-pod my-app --protocol dns

# Sample output:
# Jan  1 12:00:01.234 [production/my-app] to [kube-system/coredns]:
#   DNS Query: api.stripe.com. AAAA
# Jan  1 12:00:01.267 [kube-system/coredns] to [production/my-app]:
#   DNS Answer: api.stripe.com. AAAA -> 2a05:d014:275:cb01::a4

HTTP Visibility

For HTTP/1.x and HTTP/2 traffic with layer 7 policy enabled:

1
2
3
4
5
6
7
8


# HTTP flows with method, URL, status
hubble observe --protocol http --follow

# Sample output:
# [production/frontend] → [production/backend]:
#   HTTP GET /api/users -> 200 OK (1.2ms)
# [production/frontend] → [production/backend]:
#   HTTP POST /api/orders -> 503 Service Unavailable (45.1ms)

Identifying Dropped Traffic

The most common debugging scenario: something can’t connect and you don’t know why.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Watch for drops in real time
hubble observe --verdict DROPPED --follow

# Sample output showing a NetworkPolicy drop:
# [production/frontend] → [production/database]:
#   TCP Flags: SYN  DROPPED  Policy denied
#   source: production/frontend (app=frontend)
#   destination: production/database (app=postgres)
#   reason: Policy denied

# More detail on what policy caused the drop
hubble observe --verdict DROPPED --output json | \
  jq '{src: .source.namespace+"/"+.source.pod_name,
       dst: .destination.namespace+"/"+.destination.pod_name,
       reason: .drop_reason_desc,
       policy: .traffic_direction}'

Time-Range Queries

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Flows from the last 5 minutes
hubble observe --since 5m

# Flows in a specific time window
hubble observe --since "2026-03-26T10:00:00Z" --until "2026-03-26T10:05:00Z"

# Count drops by destination in the last hour
hubble observe --verdict DROPPED --since 1h --output json | \
  jq -r '.destination.namespace + "/" + .destination.pod_name' | \
  sort | uniq -c | sort -rn | head -20

Hubble UI: The Service Map

The Hubble UI renders a live dependency graph of your services. Access it:

1
2
3
4
5


# Port-forward the UI
kubectl port-forward -n kube-system svc/hubble-ui 12000:80

# Or with cilium CLI
cilium hubble ui

The UI shows:

Service dependency map: which services are talking to which, rendered as a graph
Flow table: clickable flows with full metadata
Namespace filter: scope the view to a specific namespace
Verdict filter: show only dropped or forwarded flows
Live mode: real-time updates as flows occur

This is invaluable when onboarding a new service — you can visually confirm it’s only talking to what it should be.

Network Policy with Cilium

Cilium supports standard Kubernetes NetworkPolicy objects and extends them with CiliumNetworkPolicy (CNP) CRDs that add L7 visibility and more powerful selectors.

Standard NetworkPolicy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


# Allow frontend to reach backend on port 8080
# Deny all other ingress to backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

With Cilium, this is enforced by eBPF — not iptables. And every dropped packet creates a Hubble flow event with the reason Policy denied.

CiliumNetworkPolicy: Layer 7 Policy

Standard NetworkPolicy works at L3/L4. CiliumNetworkPolicy adds L7 rules:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


# Allow GET /api/public but deny POST /api/admin
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-http-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api-server
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: GET
                path: /api/public
              - method: POST
                path: /api/orders
                # Only allow requests with valid auth header
                headers:
                  - Authorization: Bearer.*

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


# DNS-based egress policy — allow only specific external destinations
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: egress-dns-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: payment-service
  egress:
    # Allow DNS resolution
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*.stripe.com"
              - matchPattern: "*.stripe.network"
    # Allow only resolved Stripe addresses
    - toFQDNs:
        - matchPattern: "*.stripe.com"
        - matchPattern: "*.stripe.network"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

The toFQDNs selector is powerful — Cilium intercepts DNS responses and dynamically adds the resolved IPs to the policy. No manual CIDR management needed.

Cluster-Wide Policy with CiliumClusterwideNetworkPolicy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# Baseline: deny all cross-namespace traffic by default
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: deny-cross-namespace
spec:
  endpointSelector: {}   # Apply to all endpoints
  ingress:
    - fromEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
    # Allow same-namespace traffic
    - fromRequires:
        - matchExpressions:
            - key: k8s:io.kubernetes.pod.namespace
              operator: In
              values:
                - "$(k8s:io.kubernetes.pod.namespace)"

Debugging Network Policy Issues

Step-by-Step Debug Workflow

Scenario: frontend can’t reach backend in the production namespace.

Step 1: Confirm the connection is failing

1
2
3
4


# Exec into frontend pod and try to connect
kubectl exec -it -n production deploy/frontend -- \
  curl -v http://backend:8080/health --connect-timeout 5
# Output: Connection timed out or Connection refused

Step 2: Watch Hubble for drops

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# In another terminal, watch for drops
hubble observe \
  --from-pod production/frontend \
  --to-pod production/backend \
  --verdict DROPPED \
  --follow

# Then retry the curl — you should see:
# [production/frontend] → [production/backend]:
#   TCP Flags: SYN  DROPPED  Policy denied

Step 3: Check the policy on the destination

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# What policies apply to the backend pod?
cilium policy get

# Check the endpoint's policy
kubectl -n kube-system exec -it ds/cilium -- \
  cilium endpoint list

# Get the endpoint ID for backend
kubectl -n kube-system exec -it ds/cilium -- \
  cilium endpoint list | grep backend

# Inspect its policy
kubectl -n kube-system exec -it ds/cilium -- \
  cilium endpoint get <ENDPOINT_ID>

Step 4: Use cilium policy trace

This is the most powerful tool — it simulates a policy decision without sending actual traffic:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Would traffic from frontend to backend on port 8080 be allowed?
kubectl -n kube-system exec -it ds/cilium -- \
  cilium policy trace \
    --src-k8s-pod production/frontend-abc123 \
    --dst-k8s-pod production/backend-xyz789 \
    --dport 8080/TCP

# Output:
# Resolving ingress policy for [production/backend-xyz789]:
# * Rule {"matchLabels":{"app":"backend"}}: selected
# * Found no allow rule. Dropped!
#
# Final verdict: DROPPED

Step 5: Fix the policy and verify

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Apply the corrected NetworkPolicy
kubectl apply -f fixed-policy.yaml

# Verify with policy trace
kubectl -n kube-system exec -it ds/cilium -- \
  cilium policy trace \
    --src-k8s-pod production/frontend-abc123 \
    --dst-k8s-pod production/backend-xyz789 \
    --dport 8080/TCP

# Output should now say: Final verdict: FORWARDED

Common Policy Pitfalls

Pitfall 1: Default deny without explicit allow for DNS

If you apply a default-deny egress policy, pods can’t resolve DNS:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# Wrong — blocks DNS
spec:
  egress: []   # or no egress section with policyTypes: [Egress]

# Right — always allow DNS
spec:
  egress:
    - toEndpoints:
        - matchLabels:
            k8s-app: kube-dns
            k8s:io.kubernetes.pod.namespace: kube-system
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY

Pitfall 2: Missing namespace selector

podSelector alone doesn’t restrict by namespace:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


# Wrong — matches any pod with app=frontend in ANY namespace
ingress:
  - from:
      - podSelector:
          matchLabels:
            app: frontend

# Right — restrict to specific namespace
ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: production
        podSelector:
          matchLabels:
            app: frontend

Pitfall 3: AND vs OR in from rules

Multiple entries in the from array are OR’d. Multiple selectors within one entry are AND’d:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# This means: (namespace=prod AND app=frontend) OR (namespace=staging)
ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            name: production
        podSelector:          # AND with namespace selector
          matchLabels:
            app: frontend
      - namespaceSelector:   # OR — separate list item
          matchLabels:
            name: staging

Hubble Metrics and Grafana Dashboards

Hubble exports rich Prometheus metrics when enabled. Here’s what’s available and how to use it.

Key Metrics

1
2
3


# View available Hubble metrics
kubectl port-forward -n kube-system ds/cilium 9965:9965 &
curl -s http://localhost:9965/metrics | grep "^# HELP hubble"

Important metrics:

Metric	Description
`hubble_flows_processed_total`	Total flows by verdict, protocol, direction
`hubble_drop_total`	Dropped packets by reason and direction
`hubble_tcp_flags_total`	TCP flag distribution (SYN, FIN, RST)
`hubble_dns_queries_total`	DNS queries by type
`hubble_dns_responses_total`	DNS responses, including NXDOMAINs
`hubble_http_requests_total`	HTTP requests by method, status, service
`hubble_http_request_duration_seconds`	HTTP latency histogram

Prometheus Scrape Config

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# If using kube-prometheus-stack, add these scrape configs:
additionalScrapeConfigs:
  - job_name: cilium-agent
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [kube-system]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_k8s_app]
        action: keep
        regex: cilium
      - source_labels: [__address__]
        action: replace
        regex: (.+):.*
        replacement: $1:9965
        target_label: __address__

  - job_name: hubble
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [kube-system]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_k8s_app]
        action: keep
        regex: hubble
      - source_labels: [__address__]
        action: replace
        regex: (.+):.*
        replacement: $1:9965
        target_label: __address__

Grafana Dashboard

Import Cilium’s official dashboards from grafana.com:

Cilium Agent Metrics — Dashboard ID: 16611
Hubble Flows — Dashboard ID: 16612
Cilium Operator — Dashboard ID: 16613

1
2
3
4


# Or deploy via ConfigMap if using grafana-operator
kubectl create configmap grafana-dashboard-cilium \
  --from-literal=dashboard.json="$(curl -s https://raw.githubusercontent.com/cilium/cilium/main/install/kubernetes/cilium/files/grafana-hubble-flows.json)" \
  -n monitoring

Useful PromQL Queries

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


# Drop rate per namespace (drops/second)
rate(hubble_drop_total[5m])

# Top namespaces by dropped traffic
topk(10,
  sum by (source_namespace) (
    rate(hubble_drop_total[5m])
  )
)

# HTTP error rate per service
sum by (destination_workload, response_code) (
  rate(hubble_http_requests_total{response_code=~"5.."}[5m])
)
/
sum by (destination_workload) (
  rate(hubble_http_requests_total[5m])
)

# DNS NXDOMAIN rate (indicates misconfigured apps or DNS issues)
rate(hubble_dns_responses_total{rcode="Non-Existent Domain"}[5m])

# P99 HTTP latency per service
histogram_quantile(0.99,
  sum by (destination_workload, le) (
    rate(hubble_http_request_duration_seconds_bucket[5m])
  )
)

# TCP reset rate (can indicate connection issues)
rate(hubble_tcp_flags_total{flags="RST"}[5m])

Alerting Rules

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cilium-alerts
  namespace: monitoring
spec:
  groups:
    - name: cilium.network
      rules:
        - alert: HighDropRate
          expr: |
            sum by (source_namespace, destination_namespace) (
              rate(hubble_drop_total[5m])
            ) > 10
          for: 2m
          labels:
            severity: warning
          annotations:
            summary: "High drop rate between {{ $labels.source_namespace }} and {{ $labels.destination_namespace }}"
            description: "{{ $value | humanize }} drops/sec — possible misconfigured NetworkPolicy"

        - alert: HTTPHighErrorRate
          expr: |
            sum by (destination_workload) (
              rate(hubble_http_requests_total{response_code=~"5.."}[5m])
            )
            /
            sum by (destination_workload) (
              rate(hubble_http_requests_total[5m])
            ) > 0.05
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "High HTTP 5xx rate on {{ $labels.destination_workload }}"
            description: "{{ $value | humanizePercentage }} error rate"

        - alert: DNSNXDOMAINSpike
          expr: |
            rate(hubble_dns_responses_total{rcode="Non-Existent Domain"}[5m]) > 5
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "High NXDOMAIN rate"
            description: "Pods may be making DNS queries for nonexistent names"

Advanced: Exporting Flows to a SIEM

For security and compliance, you may need to export all network flows to a SIEM or log aggregator. Hubble provides a gRPC API that can stream all flows.

Hubble Export to JSON

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Stream all flows as JSON (useful for piping to log shippers)
hubble observe --output json --follow | \
  jq '{
    time: .time,
    src_namespace: .source.namespace,
    src_pod: .source.pod_name,
    dst_namespace: .destination.namespace,
    dst_pod: .destination.pod_name,
    verdict: .verdict,
    protocol: .l4 | keys[0],
    drop_reason: .drop_reason_desc
  }' >> /var/log/hubble-flows.json

Hubble Export Sidecar

For production use, deploy a Hubble exporter that ships flows to Elasticsearch, Splunk, or S3:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


# DaemonSet that runs hubble observe on each node and ships to stdout (for Fluentd/Fluent Bit)
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: hubble-exporter
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: hubble-exporter
  template:
    metadata:
      labels:
        app: hubble-exporter
    spec:
      containers:
        - name: hubble-exporter
          image: quay.io/cilium/hubble:latest
          command:
            - hubble
            - observe
            - --server=unix:///var/run/cilium/hubble.sock
            - --output=json
            - --follow
          volumeMounts:
            - name: cilium-run
              mountPath: /var/run/cilium
      volumes:
        - name: cilium-run
          hostPath:
            path: /var/run/cilium
            type: Directory

Cilium’s Built-in Flow Export

Cilium 1.15+ supports built-in flow export configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


# helm values
hubble:
  export:
    static:
      enabled: true
      filePath: /var/run/cilium/hubble/events.log
      fieldMask:
        - time
        - source
        - destination
        - verdict
        - drop_reason_desc
        - traffic_direction
      allowList:
        - verdict: ["DROPPED", "ERROR"]   # Only export drops

Security Use Cases

Cilium and Hubble aren’t just for debugging — they’re a complete network security platform.

Detecting Lateral Movement

1
2
3
4
5
6
7
8
9


# Watch for any pod trying to connect to unusual ports
hubble observe \
  --verdict FORWARDED \
  --not --port 80 --not --port 443 --not --port 53 \
  --not --port 5432 --not --port 6379 \   # Your known legitimate ports
  --follow

# Or as a Prometheus alert: unexpected port usage spike
rate(hubble_flows_processed_total{destination_port!~"80|443|53|5432|6379"}[5m]) > 0

Detecting DNS Exfiltration

1
2
3
4
5
6


# Large numbers of DNS queries from a single pod
hubble observe --protocol dns --output json --since 1h | \
  jq -r '.source.pod_name' | sort | uniq -c | sort -rn | head -10

# Or in PromQL:
topk(5, rate(hubble_dns_queries_total[5m]))

Enforcing Zero-Trust with Policy Audit Mode

Before switching to full enforcement, run in audit mode to see what would be denied:

1
2
3
4
5
6


# Enable policy audit mode globally
kubectl annotate namespace production \
  policy.cilium.io/proxy-visibility=Audit

# Watch what would have been dropped
hubble observe --verdict AUDIT --follow

Quick Reference

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


# Install Cilium CLI
# https://github.com/cilium/cilium-cli/releases/latest

# Install Hubble CLI
# https://github.com/cilium/hubble/releases/latest

# Check cluster status
cilium status

# Run connectivity test
cilium connectivity test

# Port-forward Hubble relay
cilium hubble port-forward &

# Stream all flows
hubble observe --follow

# Stream drops only
hubble observe --verdict DROPPED --follow

# Debug specific connection
hubble observe --from-pod ns/pod-a --to-pod ns/pod-b --follow

# Simulate policy decision
kubectl -n kube-system exec -it ds/cilium -- \
  cilium policy trace \
    --src-k8s-pod ns/pod-a \
    --dst-k8s-pod ns/pod-b \
    --dport 8080/TCP

# Open Hubble UI
cilium hubble ui

# List all endpoints and their policy status
kubectl -n kube-system exec -it ds/cilium -- cilium endpoint list

# Check a specific endpoint's policy
kubectl -n kube-system exec -it ds/cilium -- cilium endpoint get <ID>

Where to Go Next

Cilium documentation: docs.cilium.io — comprehensive, well-maintained
Cilium Service Mesh: Cilium can replace traditional sidecar service meshes using eBPF, eliminating per-pod Envoy proxies
Tetragon: Cilium’s security observability tool — goes beyond network flows to kernel-level process execution, file access, and privilege escalation events
Cilium BGP Control Plane: use Cilium to peer with your datacenter BGP fabric for LoadBalancer IP advertisement
Cilium Gateway API: native Kubernetes Gateway API implementation using Envoy, replacing traditional ingress controllers

The combination of Cilium and Hubble shifts network observability from reactive (something broke, now I debug) to proactive (I can see everything, I can alert on anomalies, I can enforce policy based on identity rather than IP). Once you’ve operated a cluster with Hubble, going back to raw iptables and silent packet drops feels like debugging with a blindfold on.