LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

Runtime Security with Falco: Syscall-Level Threat Detection for Kubernetes

securityfalcokubernetesruntime-securitythreat-detectionebpfdevsecops

Runtime Security with Falco: Syscall-Level Threat Detection for Kubernetes

Container security tools like image scanners (Trivy, Grype) and admission controllers (OPA Gatekeeper, Kyverno) are essential — but they only defend against known bad configurations at deploy time. They can’t tell you what’s happening right now inside a running container.

That’s the gap Falco fills. It sits at the Linux kernel level, watching every system call made by every process on the host. When a container that should only be serving HTTP requests suddenly calls execve to spawn a shell, or opens /etc/passwd for writing, or establishes a network connection to an unexpected IP — Falco sees it in microseconds and fires an alert.

This guide covers how Falco works under the hood, the rule language, writing effective custom rules, integrating with your alerting stack, and building a complete runtime security posture for production Kubernetes clusters.


How Falco Works: The Kernel Driver

Falco’s core insight is that containers are just Linux processes — they make the same system calls as any other process. If you can observe syscalls, you can observe everything a container does regardless of what language it’s written in or what runtime it uses.

Three driver options

┌─────────────────────────────────────────────────────────┐
│  User Space                                             │
│  ┌─────────────────────────────────────────────────┐   │
│  │  falco (rules engine + alerting)                │   │
│  └───────────────────┬─────────────────────────────┘   │
│                      │ reads events                     │
└──────────────────────┼─────────────────────────────────┘
                       │
┌──────────────────────┼─────────────────────────────────┐
│  Kernel Space        │                                  │
│          ┌───────────▼────────────┐                    │
│          │  Falco driver (choose  │                    │
│          │  one):                 │                    │
│          │  • Kernel module (.ko) │  ← classic         │
│          │  • eBPF probe          │  ← recommended     │
│          │  • Modern eBPF (CO-RE) │  ← best            │
│          └───────────┬────────────┘                    │
│                      │ intercepts syscalls              │
│  ┌───────────────────▼────────────────────────────┐   │
│  │  Linux kernel syscall table                    │   │
│  └────────────────────────────────────────────────┘   │
└────────────────────────────────────────────────────────┘

Kernel module: Loaded as a .ko file. Fast, but requires --privileged or CAP_SYS_MODULE. Breaks on kernel upgrades.

Legacy eBPF probe: A compiled eBPF program. Safer than a kernel module. Requires kernel 4.14+.

Modern eBPF (CO-RE): Uses BTF (BPF Type Format) for kernel-version-independent eBPF. No precompiled artifacts needed. Requires kernel 5.8+. This is the recommended choice for new deployments.

What Falco observes

Falco receives a stream of enriched events for every syscall. Each event includes:

  • The syscall name (open, execve, connect, ptrace, …)
  • All arguments (file path opened, process being exec’d, destination IP, …)
  • Process metadata: pid, ppid, name, exe path, args, user, container ID
  • Container metadata: image, name, labels (fetched from the container runtime)
  • Kubernetes metadata: pod name, namespace, labels, deployment name (via k8s API)

This enriched context is what makes Falco rules expressive — you’re not just seeing “a file was opened,” you’re seeing “the process bash (pid 12345, ppid 12344: python) inside pod frontend-7d9f8b-xkp2l (namespace: production, deployment: frontend) opened /etc/shadow.”


Installing Falco on Kubernetes

Helm installation (modern eBPF driver)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=modern_ebpf \
  --set collectors.kubernetes.enabled=true \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true \
  --values falco-values.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
# falco-values.yaml
driver:
  kind: modern_ebpf  # Requires kernel >= 5.8

# Fetch Kubernetes metadata for enriched events
collectors:
  kubernetes:
    enabled: true

# Falco configuration
falco:
  # JSON output for easy parsing by log shippers
  json_output: true
  json_include_output_property: true

  # Log level for Falco's own logs (not rule output)
  log_level: info

  # Priority threshold — only alert on rules at this level or above
  # During initial deployment, use "notice" to reduce noise
  priority: notice

  # Output channels
  stdout_output:
    enabled: true

  # gRPC output for falcosidekick
  grpc_output:
    enabled: true

  grpc:
    enabled: true
    bind_address: "unix:///run/falco/falco.sock"

# Falcosidekick — fan-out alerts to Slack, PagerDuty, etc.
falcosidekick:
  enabled: true
  config:
    slack:
      webhookurl: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
      minimumpriority: "warning"
    pagerduty:
      routingKey: "YOUR_PAGERDUTY_ROUTING_KEY"
      minimumpriority: "critical"
    prometheus:
      # Expose Prometheus metrics for alert counts
      enabled: true
  webui:
    enabled: true
    replicaCount: 1

Verify installation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Check Falco pods are running
kubectl get pods -n falco

# Tail Falco's output
kubectl logs -n falco daemonset/falco -f | jq '.'

# Trigger a test alert — run a shell inside a running pod
kubectl exec -n default deploy/nginx -- sh -c 'cat /etc/shadow'
# Falco should immediately output an alert like:
# {"output":"Warning Sensitive file opened for reading by non-trusted program
#   (user=root user_loginuid=-1 program=cat command=cat /etc/shadow ...)",...}

Understanding Falco Rules

Rules are the heart of Falco. Each rule defines a condition (what to detect) and output (what to log).

Rule anatomy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
# A complete rule with all fields explained
- rule: Shell Spawned in Container
  desc: >
    A shell was spawned inside a container. Containers should not run interactive
    shells — this often indicates a breakout attempt or unauthorized access.
  condition: >
    spawned_process
    and container
    and shell_procs
    and not proc.pname in (shell_spawning_containers)
  output: >
    Shell spawned in container
    (user=%user.name user_loginuid=%user.loginuid
     container_id=%container.id container_name=%container.name
     image=%container.image.repository:%container.image.tag
     shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline
     pod=%k8s.pod.name ns=%k8s.ns.name)
  priority: WARNING
  tags: [container, shell, mitre_execution]

Fields:

  • rule: Unique name
  • desc: Human-readable description for alert consumers
  • condition: Boolean expression over Falco fields and macros
  • output: Log message with %field.name interpolation
  • priority: EMERGENCY, ALERT, CRITICAL, ERROR, WARNING, NOTICE, INFORMATIONAL, DEBUG
  • tags: MITRE ATT&CK categories, environment tags, etc.

Macros: reusable condition fragments

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
# Macros are building blocks — define once, reuse everywhere
- macro: container
  condition: container.id != host

- macro: spawned_process
  condition: evt.type = execve and evt.dir = <

- macro: shell_procs
  condition: proc.name in (bash, sh, zsh, ksh, fish, tcsh, csh, dash)

- macro: network_connection
  condition: evt.type in (connect, accept) and evt.dir = <

- macro: outbound
  condition: >
    network_connection
    and fd.typechar = 4  # IPv4
    and fd.sip != "0.0.0.0"
    and not fd.sip in (rfc_1918_addresses)

Lists: named collections of values

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# Lists allow DRY rule writing
- list: shell_spawning_containers
  items: [gitlab-runner, jenkins, ansible, docker-in-docker]

- list: trusted_image_prefixes
  items:
    - myorg/
    - gcr.io/myproject/

- list: known_outbound_destinations
  items:
    - "10.0.0.0/8"
    - "172.16.0.0/12"
    - "192.168.0.0/16"

- list: sensitive_files
  items:
    - /etc/shadow
    - /etc/passwd
    - /etc/sudoers
    - /root/.ssh
    - /proc/*/mem
    - /var/run/docker.sock

- list: crypto_mining_ports
  items: [3333, 4444, 5555, 7777, 8333, 14433, 45560]

The Falco Field Catalog

Conditions and outputs use fields from several namespaces. The most useful:

Process fields

Field Description Example
proc.name Process name nginx, python3
proc.exe Full executable path /usr/bin/python3
proc.cmdline Full command line python3 -c 'import os; os.system("id")'
proc.pid Process ID 1234
proc.pname Parent process name bash
proc.pcmdline Parent command line bash -i
proc.aname[2] Ancestor process name at depth 2 containerd-shim
user.name Effective username root, www-data
user.uid Effective UID 0, 1000

File/FD fields

Field Description
fd.name File descriptor name (path for files, IP:port for sockets)
fd.typechar f=file, 4=IPv4, 6=IPv6, u=Unix socket
fd.sip Server IP (for network events)
fd.sport Server port
fd.cip Client IP
evt.rawres Syscall return value (0=success, negative=error)

Container fields

Field Description
container.id Container short ID
container.name Container name
container.image.repository Image name (without tag)
container.image.tag Image tag
container.privileged true if --privileged

Kubernetes fields

Field Description
k8s.pod.name Pod name
k8s.ns.name Namespace
k8s.pod.label[app] Pod label value
k8s.deployment.name Owning Deployment
k8s.daemonset.name Owning DaemonSet

Writing Custom Rules

Rule 1: Detect cryptocurrency mining

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
- list: crypto_mining_domains
  items:
    - pool.minexmr.com
    - xmrpool.eu
    - supportxmr.com
    - moneropool.com
    - nanopool.org

- list: crypto_mining_ports
  items: [3333, 4444, 5555, 7777, 8333, 9999, 14433, 45560]

- rule: Cryptocurrency Mining Detected
  desc: >
    Process is connecting to known cryptocurrency mining pool ports or domains.
    This almost always indicates compromised workload or supply-chain attack.
  condition: >
    outbound
    and (fd.sport in (crypto_mining_ports)
         or fd.sip.name in (crypto_mining_domains))
  output: >
    Possible cryptocurrency mining outbound connection
    (user=%user.name container=%container.name image=%container.image.repository
     connection=%fd.name cmd=%proc.cmdline pod=%k8s.pod.name ns=%k8s.ns.name)
  priority: CRITICAL
  tags: [network, cryptomining, mitre_impact]

Rule 2: Detect container escape attempts

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# Mounting the host filesystem is a common escape vector
- rule: Container Mounting Host Filesystem
  desc: >
    A process is attempting to mount the host filesystem inside a container.
    This is a common container escape technique.
  condition: >
    evt.type = mount
    and container
    and evt.arg.flags contains MS_BIND
    and (evt.arg.dev startswith /dev/
         or evt.arg.dir = /
         or evt.arg.dir startswith /host)
  output: >
    Container attempting to mount host filesystem
    (user=%user.name container=%container.name image=%container.image.repository
     device=%evt.arg.dev mountpoint=%evt.arg.dir flags=%evt.arg.flags
     pod=%k8s.pod.name ns=%k8s.ns.name)
  priority: CRITICAL
  tags: [container, escape, mitre_privilege_escalation]

# Accessing the Docker socket = full host control
- rule: Docker Socket Access
  desc: >
    A process is accessing the Docker Unix socket. This grants the container
    full control over the Docker daemon and effectively root on the host.
  condition: >
    open_write
    and container
    and fd.name = /var/run/docker.sock
  output: >
    Docker socket write access from container
    (user=%user.name container=%container.name image=%container.image.repository
     cmd=%proc.cmdline pod=%k8s.pod.name ns=%k8s.ns.name)
  priority: CRITICAL
  tags: [container, escape, mitre_privilege_escalation]

# nsenter into host namespaces
- rule: Namespace Escape via nsenter
  desc: A process is using nsenter to access host namespaces.
  condition: >
    spawned_process
    and proc.name = nsenter
    and container
  output: >
    nsenter called inside container
    (user=%user.name container=%container.name image=%container.image.repository
     cmdline=%proc.cmdline pod=%k8s.pod.name ns=%k8s.ns.name)
  priority: CRITICAL
  tags: [container, escape, mitre_privilege_escalation]

Rule 3: Detect data exfiltration patterns

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# Unexpected outbound data to the internet
- macro: expected_outbound_services
  condition: >
    fd.sip in (known_outbound_destinations)
    or fd.sport in (80, 443)  # HTTP/HTTPS to any destination for web-facing services

- rule: Unexpected Outbound Network Connection
  desc: >
    Container established an outbound network connection to an unexpected
    destination. Could indicate C2 communication, data exfiltration, or
    supply-chain attack phoning home.
  condition: >
    outbound
    and container
    and not expected_outbound_services
    and not proc.name in (curl, wget, apt, yum, pip)  # Package managers OK during builds
    and not k8s.ns.name in (kube-system, monitoring)
  output: >
    Unexpected outbound connection from container
    (user=%user.name proc=%proc.name cmdline=%proc.cmdline
     connection=%fd.name container=%container.name image=%container.image.repository
     pod=%k8s.pod.name ns=%k8s.ns.name)
  priority: WARNING
  tags: [network, exfiltration, mitre_exfiltration]

# Reading sensitive credential files
- rule: Credential File Read
  desc: >
    A process read a file commonly containing credentials or secrets.
    In containers, this may indicate an attacker harvesting credentials
    for lateral movement.
  condition: >
    open_read
    and container
    and fd.name in (sensitive_files)
    and not proc.name in (grep, awk, sed, cat)  # Allow during init/healthcheck
    and not k8s.ns.name in (falco, kube-system)
  output: >
    Sensitive file read in container
    (user=%user.name proc=%proc.name file=%fd.name
     container=%container.name image=%container.image.repository
     pod=%k8s.pod.name ns=%k8s.ns.name)
  priority: WARNING
  tags: [filesystem, credentials, mitre_credential_access]

Rule 4: Kubernetes API server abuse

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# ServiceAccount token being used from within a container
- rule: K8s ServiceAccount Token Read
  desc: >
    A process is reading the Kubernetes service account token. Legitimate
    applications read it once at startup. Repeated reads may indicate
    an attacker harvesting credentials for cluster lateral movement.
  condition: >
    open_read
    and container
    and fd.name startswith /var/run/secrets/kubernetes.io/serviceaccount/
    and not proc.name in (python, python3, java, node, ruby, go)
    and not proc.cmdline contains "healthcheck"
  output: >
    K8s service account token read by unexpected process
    (user=%user.name proc=%proc.name cmdline=%proc.cmdline
     file=%fd.name container=%container.name image=%container.image.repository
     pod=%k8s.pod.name ns=%k8s.ns.name)
  priority: NOTICE
  tags: [kubernetes, credentials, mitre_credential_access]

# Modification of sensitive Kubernetes resources
- rule: K8s ClusterRole Binding Modified
  desc: >
    A ClusterRoleBinding was created or modified. This could grant elevated
    privileges to a service account — a common persistence mechanism after
    initial compromise.
  condition: >
    ka.verb in (create, update, patch)
    and ka.target.resource = clusterrolebindings
    and not ka.user.name in (system:serviceaccount:kube-system:clusterrole-aggregation-controller)
  output: >
    ClusterRoleBinding modified
    (user=%ka.user.name verb=%ka.verb resource=%ka.target.resource
     name=%ka.target.name response=%ka.response.code)
  priority: WARNING
  source: k8s_audit
  tags: [kubernetes, rbac, mitre_persistence]

Rule 5: Process injection and privilege escalation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# ptrace is used for debuggers but also for process injection
- rule: Ptrace Anti-Debug or Injection Attempt
  desc: >
    A process called ptrace against another process. In production containers
    this is almost always malicious — used for process injection or
    sandbox escape.
  condition: >
    evt.type = ptrace
    and evt.dir = <
    and container
    and evt.arg.pid != 0
    and not proc.name in (strace, gdb, lldb)
  output: >
    Ptrace called on process in container
    (user=%user.name proc=%proc.name target_pid=%evt.arg.pid
     container=%container.name image=%container.image.repository
     pod=%k8s.pod.name ns=%k8s.ns.name)
  priority: WARNING
  tags: [process, injection, mitre_defense_evasion]

# Setuid/setgid — privilege escalation
- rule: Setuid or Setgid Called in Container
  desc: >
    A process dropped or escalated privileges via setuid/setgid.
    Unexpected privilege changes in containers are a red flag.
  condition: >
    evt.type in (setuid, setgid)
    and container
    and evt.arg.uid != 65534  # nobody
    and user.uid != 0
    and evt.rawres = 0  # Succeeded
  output: >
    Privilege change via setuid/setgid in container
    (user=%user.name proc=%proc.name uid=%evt.arg.uid
     container=%container.name image=%container.image.repository
     pod=%k8s.pod.name ns=%k8s.ns.name)
  priority: WARNING
  tags: [process, privilege_escalation, mitre_privilege_escalation]

Tuning Rules: Reducing False Positives

Out of the box, Falco’s default ruleset generates noise. Systematic tuning is essential before you can trust alerts.

Strategy 1: Override rules with append and override

1
2
3
4
5
6
# Add exceptions to a built-in rule without replacing the whole thing
- rule: Write below binary dir
  override:
    condition: append
  # Append to the existing condition with AND NOT
  condition: and not proc.name in (npm, yarn, pip, pip3) and not k8s.ns.name = ci-builds

Strategy 2: Scoped exceptions with exceptions

The modern way to tune built-in rules:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
- rule: Read sensitive file untrusted
  exceptions:
    # Exception name, fields to match, and values
    - name: trusted_readers
      fields: [proc.name, fd.name]
      comps: [in, in]
      values:
        - [[filebeat, fluentd], [/etc/passwd, /etc/group]]
    - name: init_containers
      fields: [k8s.pod.label[lifecycle]]
      comps: [=]
      values:
        - [init]

Strategy 3: Environment-based tuning

1
2
3
4
5
6
7
8
# Only fire in production; be less strict in staging
- rule: Unexpected Outbound Connection
  condition: >
    outbound
    and container
    and not expected_outbound_services
    and k8s.pod.label[environment] = production  # Only alert in prod
  priority: WARNING

Strategy 4: Use NOTICE priority during initial rollout

Lower-priority alerts are still logged but won’t page anyone:

1
2
3
4
5
# Demote noisy rules during initial tuning period
- rule: Shell Spawned in Container
  override:
    priority: replace
  priority: NOTICE  # Demoted from WARNING — won't trigger PagerDuty

Workflow for reducing false positives

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# 1. Run Falco in dry-run / logging-only mode for a week
#    Review which rules fire most
kubectl logs -n falco daemonset/falco -f | \
  jq -r '.rule' | sort | uniq -c | sort -rn | head -20

# 2. For the noisiest rules, inspect the full events
kubectl logs -n falco daemonset/falco -f | \
  jq 'select(.rule == "Read sensitive file untrusted")' | \
  jq '{proc: .output_fields["proc.name"], file: .output_fields["fd.name"],
       image: .output_fields["container.image.repository"],
       ns: .output_fields["k8s.ns.name"]}' | \
  sort | uniq -c | sort -rn

# 3. Add exceptions for legitimate patterns found above
# 4. Promote surviving true positives to higher priority

Falcosidekick: Routing Alerts Everywhere

Falcosidekick is a fan-out forwarder that takes Falco’s gRPC output and routes it to 60+ destinations.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
# falcosidekick-config.yaml
config:
  # Slack: different channels by severity
  slack:
    webhookurl: "https://hooks.slack.com/services/..."
    channel: "#security-alerts"
    minimumpriority: notice
    messageformat: |
      :rotating_light: *{{ .Rule }}* ({{ .Priority }})
      *Pod:* `{{ index .OutputFields "k8s.pod.name" }}` in `{{ index .OutputFields "k8s.ns.name" }}`
      *Image:* `{{ index .OutputFields "container.image.repository" }}`
      *Command:* `{{ index .OutputFields "proc.cmdline" }}`
      *Time:* {{ .Time }}

  # PagerDuty: only critical alerts page on-call
  pagerduty:
    routingKey: "..."
    minimumpriority: critical

  # Elasticsearch: full audit trail
  elasticsearch:
    hostport: "http://elasticsearch:9200"
    index: falco-alerts
    minimumpriority: debug  # Store everything

  # Prometheus: metrics dashboard
  prometheus:
    enabled: true
    # Exposes: falcosidekick_falco_alerts_total{priority,rule,source,tags,hostname}

  # Webhook: custom SIEM or SOAR integration
  webhook:
    address: "https://your-siem.example.com/api/falco"
    customHeaders:
      Authorization: "Bearer ${SIEM_API_TOKEN}"
    minimumpriority: warning

  # OpsGenie: for teams using OpsGenie
  opsgenie:
    apikey: "..."
    minimumpriority: critical

  # Teams: Microsoft Teams channel
  teams:
    webhookurl: "https://outlook.office.com/webhook/..."
    minimumpriority: warning

  # Grafana annotations: mark security events on dashboards
  grafana:
    hostport: "http://grafana:3000"
    apikey: "..."
    dashboardid: 12  # Your main ops dashboard
    panelid: 1
    minimumpriority: warning

Automated response with Falcosidekick + AWS Lambda

1
2
3
4
5
6
7
# Trigger a Lambda to quarantine a compromised pod
config:
  aws:
    lambda:
      functionname: "falco-quarantine-pod"
      minimumpriority: critical
      region: us-east-1
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# lambda_function.py — automatically cordon and isolate compromised pods
import boto3
import json
from kubernetes import client, config

def lambda_handler(event, context):
    # Parse Falco alert
    alert = json.loads(event['body'])
    pod_name = alert['output_fields'].get('k8s.pod.name')
    namespace = alert['output_fields'].get('k8s.ns.name')
    rule = alert['rule']

    if not pod_name or not namespace:
        return {'statusCode': 400}

    # Only auto-quarantine critical alerts for known attack rules
    if alert['priority'] != 'Critical':
        return {'statusCode': 200, 'body': 'Non-critical, skipping'}

    print(f"QUARANTINE: {pod_name}/{namespace} due to rule: {rule}")

    # Load in-cluster config
    config.load_incluster_config()
    v1 = client.CoreV1Api()

    # Label the pod as quarantined
    v1.patch_namespaced_pod(
        name=pod_name,
        namespace=namespace,
        body={"metadata": {"labels": {"security.example.com/quarantined": "true"}}}
    )

    # NetworkPolicy will isolate pods with this label (pre-deployed)
    # The pod stays alive for forensics but can't talk to anything

    # Notify security team with forensic info
    logs = v1.read_namespaced_pod_log(pod_name, namespace, tail_lines=500)
    notify_security_team(pod_name, namespace, rule, logs)

    return {'statusCode': 200, 'body': f'Quarantined {pod_name}'}
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# NetworkPolicy: isolate quarantined pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: quarantine-isolation
  namespace: default
spec:
  podSelector:
    matchLabels:
      security.example.com/quarantined: "true"
  policyTypes:
    - Ingress
    - Egress
  # Empty ingress/egress = deny all traffic

Kubernetes Audit Log Rules

Beyond syscalls, Falco can analyze the Kubernetes audit log for control-plane level attacks.

Enable K8s audit logging

1
2
3
4
5
6
# kube-apiserver configuration
--audit-log-path=/var/log/kubernetes/audit.log
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# audit-policy.yaml — what to log
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log all requests to sensitive resources
  - level: RequestResponse
    resources:
      - group: ""
        resources: [secrets, serviceaccounts]
      - group: rbac.authorization.k8s.io
        resources: [clusterroles, clusterrolebindings, roles, rolebindings]

  # Log exec/port-forward (common post-exploitation activity)
  - level: RequestResponse
    resources:
      - group: ""
        resources: [pods/exec, pods/portforward, pods/attach]

  # Log everything else at metadata level
  - level: Metadata

K8s audit rules for Falco

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
# Detect exec into running pods (common investigation AND attack path)
- rule: Exec Pod
  desc: >
    kubectl exec was used to get a shell in a running pod.
    Legitimate for debugging but often used by attackers post-compromise.
  condition: >
    ka.verb = create
    and ka.target.subresource = exec
    and not ka.user.name startswith system:
    and not ka.user.name in (trusted_kubectl_users)
  output: >
    Exec into pod
    (user=%ka.user.name pod=%ka.target.name ns=%ka.target.namespace
     container=%ka.req.pod.containers.name command=%ka.uri.param[command])
  priority: NOTICE
  source: k8s_audit
  tags: [kubernetes, exec, mitre_execution]

# Detect RBAC escalation
- rule: K8s Secret Access
  desc: >
    A non-system user read a Kubernetes Secret. Could indicate credential
    theft or misconfigured RBAC.
  condition: >
    ka.verb in (get, list, watch)
    and ka.target.resource = secrets
    and not ka.user.name startswith system:
    and ka.response.code = 200
  output: >
    K8s Secret accessed
    (user=%ka.user.name verb=%ka.verb ns=%ka.target.namespace
     secret=%ka.target.name response=%ka.response.code)
  priority: WARNING
  source: k8s_audit
  tags: [kubernetes, secrets, mitre_credential_access]

# Anonymous access attempts
- rule: Anonymous Request Allowed
  desc: An anonymous (unauthenticated) request was allowed to the K8s API.
  condition: >
    ka.user.name = system:anonymous
    and ka.response.code = 200
  output: >
    Anonymous request allowed to K8s API
    (verb=%ka.verb resource=%ka.target.resource uri=%ka.uri)
  priority: ERROR
  source: k8s_audit
  tags: [kubernetes, authentication, mitre_initial_access]

Prometheus Metrics and Grafana Dashboard

Falcosidekick exposes metrics for building a security dashboard:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Alert rate by priority
sum(rate(falcosidekick_falco_alerts_total[5m])) by (priority)

# Alert rate by rule
topk(10, sum(rate(falcosidekick_falco_alerts_total[1h])) by (rule))

# Critical alerts in last hour (should alert on this)
increase(falcosidekick_falco_alerts_total{priority="Critical"}[1h])

# Alert volume by namespace
sum(rate(falcosidekick_falco_alerts_total[5m])) by (hostname)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# Prometheus alert: critical Falco alert received
groups:
  - name: falco_security
    rules:
      - alert: FalcoCriticalAlert
        expr: increase(falcosidekick_falco_alerts_total{priority="Critical"}[5m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "Falco critical security alert: {{ $labels.rule }}"
          runbook_url: "https://runbooks.example.com/falco-critical"

      - alert: FalcoHighAlertVolume
        expr: sum(rate(falcosidekick_falco_alerts_total[5m])) > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High Falco alert volume ({{ $value }}/s) — possible attack or rule tuning needed"

      - alert: FalcoAgentDown
        expr: up{job="falco"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Falco agent down on {{ $labels.instance }} — runtime security is blind"

Falco in CI/CD: Detecting Malicious Build Behavior

Run Falco during CI builds to detect supply-chain attacks (a compromised dependency running code during npm install):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# GitHub Actions: run Falco alongside the build
jobs:
  secure-build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Start Falco
        run: |
          # Install Falco in syscall capture mode (no K8s)
          curl -fsSL https://falco.org/install.sh | bash
          sudo falco \
            --modern-bpf \
            --json-output \
            --rules-file /etc/falco/falco_rules.yaml \
            --rules-file ./ci-falco-rules.yaml \
            > /tmp/falco-output.jsonl &
          echo "FALCO_PID=$!" >> $GITHUB_ENV

      - name: Build (under Falco observation)
        run: |
          npm ci
          npm run build
          npm test

      - name: Check Falco alerts
        if: always()
        run: |
          kill $FALCO_PID || true
          # Fail the build if any WARNING+ alerts fired during build
          CRITICAL=$(cat /tmp/falco-output.jsonl | \
            jq 'select(.priority == "Critical" or .priority == "Error" or .priority == "Warning")' | \
            wc -l)
          if [ "$CRITICAL" -gt 0 ]; then
            echo "::error::Falco detected suspicious activity during build!"
            cat /tmp/falco-output.jsonl | jq 'select(.priority != "Debug" and .priority != "Informational")'
            exit 1
          fi
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# ci-falco-rules.yaml — rules specific to build environments
- rule: Build Process Network Exfiltration
  desc: >
    Build process made an unexpected network connection. Could indicate
    a compromised dependency exfiltrating code or secrets.
  condition: >
    outbound
    and not proc.name in (npm, node, pip, python, curl, wget, git, go, cargo, mvn, gradle)
    and not fd.sport in (80, 443, 22)
  output: >
    Unexpected network connection during build
    (proc=%proc.name cmdline=%proc.cmdline connection=%fd.name)
  priority: WARNING

Performance Considerations

Falco’s syscall interception has a measurable but manageable overhead:

Driver Typical CPU overhead Notes
Kernel module 1–3% Fastest interception path
Legacy eBPF 2–4% Safer, slightly more overhead
Modern eBPF (CO-RE) 2–5% Best compatibility, recommended

To reduce overhead:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# falco.yaml — performance tuning
# Increase syscall buffer size to reduce drops under high load
syscall_buf_size_preset: 4  # Range 1-10, default 4. Higher = more memory, fewer drops

# Drop syscalls not needed by any rule (Falco does this automatically based on loaded rules)
# Explicitly set the set of interesting syscalls if you know you don't need certain ones:
base_syscalls:
  custom_set: [execve, execveat, open, openat, connect, accept, ptrace, setuid, setgid, mount]
  repair: true

# Monitor for syscall drops
metrics:
  enabled: true
  interval: 15m
  output_rule: true
  resource_utilization_enabled: true
1
2
3
4
5
6
# Check if Falco is dropping events (means host is too busy for the buffer)
kubectl logs -n falco daemonset/falco | grep "Falco internal: syscall event drop"

# If drops are occurring, increase buffer:
helm upgrade falco falcosecurity/falco --reuse-values \
  --set "falco.syscall_buf_size_preset=6"

Quick Reference: Default Rules Worth Knowing

Falco ships with a comprehensive default ruleset. The highest-value rules to understand:

Rule What it catches
Terminal shell in container Interactive shell spawned in a running container
Write below binary dir Writes to /bin, /sbin, /usr/bin etc.
Write below etc Writes to /etc in a container
Read sensitive file untrusted Reads of /etc/shadow, /etc/sudoers, SSH keys
Run shell untrusted Unexpected shell from non-shell parent process
Outbound or Inbound Traffic not to Authorized Server Process Unexpected network activity
System user interactive System accounts (www-data, nobody) running interactive sessions
Modify binary dirs Modification of system binary directories
Change thread namespace setns calls — namespace escape attempts
Launch Privileged Container docker run --privileged or equivalent
Contact K8S API Server From Container Pod calling K8s API directly
Packet socket created in container Raw socket creation — could be network sniffing

Summary

Falco gives you visibility that no other tool provides: ground truth about what is actually happening inside your running containers at the kernel level. No container runtime, no language, no obfuscation technique can hide syscalls from a properly deployed Falco instance.

The deployment path:

  1. Install with modern eBPF driver — no kernel module required, CO-RE means it survives kernel upgrades
  2. Start with default rules in NOTICE mode — observe the noise before tuning
  3. Audit what fires — identify legitimate processes triggering alerts, add exceptions
  4. Write custom rules for your threat model — what would an attacker do in your environment?
  5. Route alerts with Falcosidekick — Slack for awareness, PagerDuty for critical, Elasticsearch for retention
  6. Build automated response — NetworkPolicy quarantine for Critical alerts, Lambda/webhook for remediation
  7. Add K8s audit rules — cover the control plane, not just the data plane
  8. Integrate into CI/CD — catch supply-chain attacks during builds

The goal isn’t zero alerts — it’s a tuned ruleset where every alert represents a genuine investigation decision. When a Falco alert fires in production and you know it means something real happened, you have exactly what a security program needs: reliable signal in the noise.

Comments