Runtime Security with Falco: Syscall-Level Threat Detection for Kubernetes
Container security tools like image scanners (Trivy, Grype) and admission controllers (OPA Gatekeeper, Kyverno) are essential — but they only defend against known bad configurations at deploy time. They can’t tell you what’s happening right now inside a running container.
That’s the gap Falco fills. It sits at the Linux kernel level, watching every system call made by every process on the host. When a container that should only be serving HTTP requests suddenly calls execve to spawn a shell, or opens /etc/passwd for writing, or establishes a network connection to an unexpected IP — Falco sees it in microseconds and fires an alert.
This guide covers how Falco works under the hood, the rule language, writing effective custom rules, integrating with your alerting stack, and building a complete runtime security posture for production Kubernetes clusters.
How Falco Works: The Kernel Driver
Falco’s core insight is that containers are just Linux processes — they make the same system calls as any other process. If you can observe syscalls, you can observe everything a container does regardless of what language it’s written in or what runtime it uses.
Three driver options
┌─────────────────────────────────────────────────────────┐
│ User Space │
│ ┌─────────────────────────────────────────────────┐ │
│ │ falco (rules engine + alerting) │ │
│ └───────────────────┬─────────────────────────────┘ │
│ │ reads events │
└──────────────────────┼─────────────────────────────────┘
│
┌──────────────────────┼─────────────────────────────────┐
│ Kernel Space │ │
│ ┌───────────▼────────────┐ │
│ │ Falco driver (choose │ │
│ │ one): │ │
│ │ • Kernel module (.ko) │ ← classic │
│ │ • eBPF probe │ ← recommended │
│ │ • Modern eBPF (CO-RE) │ ← best │
│ └───────────┬────────────┘ │
│ │ intercepts syscalls │
│ ┌───────────────────▼────────────────────────────┐ │
│ │ Linux kernel syscall table │ │
│ └────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────┘
Kernel module: Loaded as a .ko file. Fast, but requires --privileged or CAP_SYS_MODULE. Breaks on kernel upgrades.
Legacy eBPF probe: A compiled eBPF program. Safer than a kernel module. Requires kernel 4.14+.
Modern eBPF (CO-RE): Uses BTF (BPF Type Format) for kernel-version-independent eBPF. No precompiled artifacts needed. Requires kernel 5.8+. This is the recommended choice for new deployments.
What Falco observes
Falco receives a stream of enriched events for every syscall. Each event includes:
- The syscall name (
open, execve, connect, ptrace, …)
- All arguments (file path opened, process being exec’d, destination IP, …)
- Process metadata: pid, ppid, name, exe path, args, user, container ID
- Container metadata: image, name, labels (fetched from the container runtime)
- Kubernetes metadata: pod name, namespace, labels, deployment name (via k8s API)
This enriched context is what makes Falco rules expressive — you’re not just seeing “a file was opened,” you’re seeing “the process bash (pid 12345, ppid 12344: python) inside pod frontend-7d9f8b-xkp2l (namespace: production, deployment: frontend) opened /etc/shadow.”
Installing Falco on Kubernetes
Helm installation (modern eBPF driver)
1
2
3
4
5
6
7
8
9
10
11
|
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco \
--namespace falco \
--create-namespace \
--set driver.kind=modern_ebpf \
--set collectors.kubernetes.enabled=true \
--set falcosidekick.enabled=true \
--set falcosidekick.webui.enabled=true \
--values falco-values.yaml
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
|
# falco-values.yaml
driver:
kind: modern_ebpf # Requires kernel >= 5.8
# Fetch Kubernetes metadata for enriched events
collectors:
kubernetes:
enabled: true
# Falco configuration
falco:
# JSON output for easy parsing by log shippers
json_output: true
json_include_output_property: true
# Log level for Falco's own logs (not rule output)
log_level: info
# Priority threshold — only alert on rules at this level or above
# During initial deployment, use "notice" to reduce noise
priority: notice
# Output channels
stdout_output:
enabled: true
# gRPC output for falcosidekick
grpc_output:
enabled: true
grpc:
enabled: true
bind_address: "unix:///run/falco/falco.sock"
# Falcosidekick — fan-out alerts to Slack, PagerDuty, etc.
falcosidekick:
enabled: true
config:
slack:
webhookurl: "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
minimumpriority: "warning"
pagerduty:
routingKey: "YOUR_PAGERDUTY_ROUTING_KEY"
minimumpriority: "critical"
prometheus:
# Expose Prometheus metrics for alert counts
enabled: true
webui:
enabled: true
replicaCount: 1
|
Verify installation
1
2
3
4
5
6
7
8
9
10
11
|
# Check Falco pods are running
kubectl get pods -n falco
# Tail Falco's output
kubectl logs -n falco daemonset/falco -f | jq '.'
# Trigger a test alert — run a shell inside a running pod
kubectl exec -n default deploy/nginx -- sh -c 'cat /etc/shadow'
# Falco should immediately output an alert like:
# {"output":"Warning Sensitive file opened for reading by non-trusted program
# (user=root user_loginuid=-1 program=cat command=cat /etc/shadow ...)",...}
|
Understanding Falco Rules
Rules are the heart of Falco. Each rule defines a condition (what to detect) and output (what to log).
Rule anatomy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
# A complete rule with all fields explained
- rule: Shell Spawned in Container
desc: >
A shell was spawned inside a container. Containers should not run interactive
shells — this often indicates a breakout attempt or unauthorized access.
condition: >
spawned_process
and container
and shell_procs
and not proc.pname in (shell_spawning_containers)
output: >
Shell spawned in container
(user=%user.name user_loginuid=%user.loginuid
container_id=%container.id container_name=%container.name
image=%container.image.repository:%container.image.tag
shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline
pod=%k8s.pod.name ns=%k8s.ns.name)
priority: WARNING
tags: [container, shell, mitre_execution]
|
Fields:
rule: Unique name
desc: Human-readable description for alert consumers
condition: Boolean expression over Falco fields and macros
output: Log message with %field.name interpolation
priority: EMERGENCY, ALERT, CRITICAL, ERROR, WARNING, NOTICE, INFORMATIONAL, DEBUG
tags: MITRE ATT&CK categories, environment tags, etc.
Macros: reusable condition fragments
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
# Macros are building blocks — define once, reuse everywhere
- macro: container
condition: container.id != host
- macro: spawned_process
condition: evt.type = execve and evt.dir = <
- macro: shell_procs
condition: proc.name in (bash, sh, zsh, ksh, fish, tcsh, csh, dash)
- macro: network_connection
condition: evt.type in (connect, accept) and evt.dir = <
- macro: outbound
condition: >
network_connection
and fd.typechar = 4 # IPv4
and fd.sip != "0.0.0.0"
and not fd.sip in (rfc_1918_addresses)
|
Lists: named collections of values
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
# Lists allow DRY rule writing
- list: shell_spawning_containers
items: [gitlab-runner, jenkins, ansible, docker-in-docker]
- list: trusted_image_prefixes
items:
- myorg/
- gcr.io/myproject/
- list: known_outbound_destinations
items:
- "10.0.0.0/8"
- "172.16.0.0/12"
- "192.168.0.0/16"
- list: sensitive_files
items:
- /etc/shadow
- /etc/passwd
- /etc/sudoers
- /root/.ssh
- /proc/*/mem
- /var/run/docker.sock
- list: crypto_mining_ports
items: [3333, 4444, 5555, 7777, 8333, 14433, 45560]
|
The Falco Field Catalog
Conditions and outputs use fields from several namespaces. The most useful:
Process fields
| Field |
Description |
Example |
proc.name |
Process name |
nginx, python3 |
proc.exe |
Full executable path |
/usr/bin/python3 |
proc.cmdline |
Full command line |
python3 -c 'import os; os.system("id")' |
proc.pid |
Process ID |
1234 |
proc.pname |
Parent process name |
bash |
proc.pcmdline |
Parent command line |
bash -i |
proc.aname[2] |
Ancestor process name at depth 2 |
containerd-shim |
user.name |
Effective username |
root, www-data |
user.uid |
Effective UID |
0, 1000 |
File/FD fields
| Field |
Description |
fd.name |
File descriptor name (path for files, IP:port for sockets) |
fd.typechar |
f=file, 4=IPv4, 6=IPv6, u=Unix socket |
fd.sip |
Server IP (for network events) |
fd.sport |
Server port |
fd.cip |
Client IP |
evt.rawres |
Syscall return value (0=success, negative=error) |
Container fields
| Field |
Description |
container.id |
Container short ID |
container.name |
Container name |
container.image.repository |
Image name (without tag) |
container.image.tag |
Image tag |
container.privileged |
true if --privileged |
Kubernetes fields
| Field |
Description |
k8s.pod.name |
Pod name |
k8s.ns.name |
Namespace |
k8s.pod.label[app] |
Pod label value |
k8s.deployment.name |
Owning Deployment |
k8s.daemonset.name |
Owning DaemonSet |
Writing Custom Rules
Rule 1: Detect cryptocurrency mining
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
- list: crypto_mining_domains
items:
- pool.minexmr.com
- xmrpool.eu
- supportxmr.com
- moneropool.com
- nanopool.org
- list: crypto_mining_ports
items: [3333, 4444, 5555, 7777, 8333, 9999, 14433, 45560]
- rule: Cryptocurrency Mining Detected
desc: >
Process is connecting to known cryptocurrency mining pool ports or domains.
This almost always indicates compromised workload or supply-chain attack.
condition: >
outbound
and (fd.sport in (crypto_mining_ports)
or fd.sip.name in (crypto_mining_domains))
output: >
Possible cryptocurrency mining outbound connection
(user=%user.name container=%container.name image=%container.image.repository
connection=%fd.name cmd=%proc.cmdline pod=%k8s.pod.name ns=%k8s.ns.name)
priority: CRITICAL
tags: [network, cryptomining, mitre_impact]
|
Rule 2: Detect container escape attempts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
|
# Mounting the host filesystem is a common escape vector
- rule: Container Mounting Host Filesystem
desc: >
A process is attempting to mount the host filesystem inside a container.
This is a common container escape technique.
condition: >
evt.type = mount
and container
and evt.arg.flags contains MS_BIND
and (evt.arg.dev startswith /dev/
or evt.arg.dir = /
or evt.arg.dir startswith /host)
output: >
Container attempting to mount host filesystem
(user=%user.name container=%container.name image=%container.image.repository
device=%evt.arg.dev mountpoint=%evt.arg.dir flags=%evt.arg.flags
pod=%k8s.pod.name ns=%k8s.ns.name)
priority: CRITICAL
tags: [container, escape, mitre_privilege_escalation]
# Accessing the Docker socket = full host control
- rule: Docker Socket Access
desc: >
A process is accessing the Docker Unix socket. This grants the container
full control over the Docker daemon and effectively root on the host.
condition: >
open_write
and container
and fd.name = /var/run/docker.sock
output: >
Docker socket write access from container
(user=%user.name container=%container.name image=%container.image.repository
cmd=%proc.cmdline pod=%k8s.pod.name ns=%k8s.ns.name)
priority: CRITICAL
tags: [container, escape, mitre_privilege_escalation]
# nsenter into host namespaces
- rule: Namespace Escape via nsenter
desc: A process is using nsenter to access host namespaces.
condition: >
spawned_process
and proc.name = nsenter
and container
output: >
nsenter called inside container
(user=%user.name container=%container.name image=%container.image.repository
cmdline=%proc.cmdline pod=%k8s.pod.name ns=%k8s.ns.name)
priority: CRITICAL
tags: [container, escape, mitre_privilege_escalation]
|
Rule 3: Detect data exfiltration patterns
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
|
# Unexpected outbound data to the internet
- macro: expected_outbound_services
condition: >
fd.sip in (known_outbound_destinations)
or fd.sport in (80, 443) # HTTP/HTTPS to any destination for web-facing services
- rule: Unexpected Outbound Network Connection
desc: >
Container established an outbound network connection to an unexpected
destination. Could indicate C2 communication, data exfiltration, or
supply-chain attack phoning home.
condition: >
outbound
and container
and not expected_outbound_services
and not proc.name in (curl, wget, apt, yum, pip) # Package managers OK during builds
and not k8s.ns.name in (kube-system, monitoring)
output: >
Unexpected outbound connection from container
(user=%user.name proc=%proc.name cmdline=%proc.cmdline
connection=%fd.name container=%container.name image=%container.image.repository
pod=%k8s.pod.name ns=%k8s.ns.name)
priority: WARNING
tags: [network, exfiltration, mitre_exfiltration]
# Reading sensitive credential files
- rule: Credential File Read
desc: >
A process read a file commonly containing credentials or secrets.
In containers, this may indicate an attacker harvesting credentials
for lateral movement.
condition: >
open_read
and container
and fd.name in (sensitive_files)
and not proc.name in (grep, awk, sed, cat) # Allow during init/healthcheck
and not k8s.ns.name in (falco, kube-system)
output: >
Sensitive file read in container
(user=%user.name proc=%proc.name file=%fd.name
container=%container.name image=%container.image.repository
pod=%k8s.pod.name ns=%k8s.ns.name)
priority: WARNING
tags: [filesystem, credentials, mitre_credential_access]
|
Rule 4: Kubernetes API server abuse
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
|
# ServiceAccount token being used from within a container
- rule: K8s ServiceAccount Token Read
desc: >
A process is reading the Kubernetes service account token. Legitimate
applications read it once at startup. Repeated reads may indicate
an attacker harvesting credentials for cluster lateral movement.
condition: >
open_read
and container
and fd.name startswith /var/run/secrets/kubernetes.io/serviceaccount/
and not proc.name in (python, python3, java, node, ruby, go)
and not proc.cmdline contains "healthcheck"
output: >
K8s service account token read by unexpected process
(user=%user.name proc=%proc.name cmdline=%proc.cmdline
file=%fd.name container=%container.name image=%container.image.repository
pod=%k8s.pod.name ns=%k8s.ns.name)
priority: NOTICE
tags: [kubernetes, credentials, mitre_credential_access]
# Modification of sensitive Kubernetes resources
- rule: K8s ClusterRole Binding Modified
desc: >
A ClusterRoleBinding was created or modified. This could grant elevated
privileges to a service account — a common persistence mechanism after
initial compromise.
condition: >
ka.verb in (create, update, patch)
and ka.target.resource = clusterrolebindings
and not ka.user.name in (system:serviceaccount:kube-system:clusterrole-aggregation-controller)
output: >
ClusterRoleBinding modified
(user=%ka.user.name verb=%ka.verb resource=%ka.target.resource
name=%ka.target.name response=%ka.response.code)
priority: WARNING
source: k8s_audit
tags: [kubernetes, rbac, mitre_persistence]
|
Rule 5: Process injection and privilege escalation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
|
# ptrace is used for debuggers but also for process injection
- rule: Ptrace Anti-Debug or Injection Attempt
desc: >
A process called ptrace against another process. In production containers
this is almost always malicious — used for process injection or
sandbox escape.
condition: >
evt.type = ptrace
and evt.dir = <
and container
and evt.arg.pid != 0
and not proc.name in (strace, gdb, lldb)
output: >
Ptrace called on process in container
(user=%user.name proc=%proc.name target_pid=%evt.arg.pid
container=%container.name image=%container.image.repository
pod=%k8s.pod.name ns=%k8s.ns.name)
priority: WARNING
tags: [process, injection, mitre_defense_evasion]
# Setuid/setgid — privilege escalation
- rule: Setuid or Setgid Called in Container
desc: >
A process dropped or escalated privileges via setuid/setgid.
Unexpected privilege changes in containers are a red flag.
condition: >
evt.type in (setuid, setgid)
and container
and evt.arg.uid != 65534 # nobody
and user.uid != 0
and evt.rawres = 0 # Succeeded
output: >
Privilege change via setuid/setgid in container
(user=%user.name proc=%proc.name uid=%evt.arg.uid
container=%container.name image=%container.image.repository
pod=%k8s.pod.name ns=%k8s.ns.name)
priority: WARNING
tags: [process, privilege_escalation, mitre_privilege_escalation]
|
Tuning Rules: Reducing False Positives
Out of the box, Falco’s default ruleset generates noise. Systematic tuning is essential before you can trust alerts.
Strategy 1: Override rules with append and override
1
2
3
4
5
6
|
# Add exceptions to a built-in rule without replacing the whole thing
- rule: Write below binary dir
override:
condition: append
# Append to the existing condition with AND NOT
condition: and not proc.name in (npm, yarn, pip, pip3) and not k8s.ns.name = ci-builds
|
Strategy 2: Scoped exceptions with exceptions
The modern way to tune built-in rules:
1
2
3
4
5
6
7
8
9
10
11
12
13
|
- rule: Read sensitive file untrusted
exceptions:
# Exception name, fields to match, and values
- name: trusted_readers
fields: [proc.name, fd.name]
comps: [in, in]
values:
- [[filebeat, fluentd], [/etc/passwd, /etc/group]]
- name: init_containers
fields: [k8s.pod.label[lifecycle]]
comps: [=]
values:
- [init]
|
Strategy 3: Environment-based tuning
1
2
3
4
5
6
7
8
|
# Only fire in production; be less strict in staging
- rule: Unexpected Outbound Connection
condition: >
outbound
and container
and not expected_outbound_services
and k8s.pod.label[environment] = production # Only alert in prod
priority: WARNING
|
Strategy 4: Use NOTICE priority during initial rollout
Lower-priority alerts are still logged but won’t page anyone:
1
2
3
4
5
|
# Demote noisy rules during initial tuning period
- rule: Shell Spawned in Container
override:
priority: replace
priority: NOTICE # Demoted from WARNING — won't trigger PagerDuty
|
Workflow for reducing false positives
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
# 1. Run Falco in dry-run / logging-only mode for a week
# Review which rules fire most
kubectl logs -n falco daemonset/falco -f | \
jq -r '.rule' | sort | uniq -c | sort -rn | head -20
# 2. For the noisiest rules, inspect the full events
kubectl logs -n falco daemonset/falco -f | \
jq 'select(.rule == "Read sensitive file untrusted")' | \
jq '{proc: .output_fields["proc.name"], file: .output_fields["fd.name"],
image: .output_fields["container.image.repository"],
ns: .output_fields["k8s.ns.name"]}' | \
sort | uniq -c | sort -rn
# 3. Add exceptions for legitimate patterns found above
# 4. Promote surviving true positives to higher priority
|
Falcosidekick: Routing Alerts Everywhere
Falcosidekick is a fan-out forwarder that takes Falco’s gRPC output and routes it to 60+ destinations.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
|
# falcosidekick-config.yaml
config:
# Slack: different channels by severity
slack:
webhookurl: "https://hooks.slack.com/services/..."
channel: "#security-alerts"
minimumpriority: notice
messageformat: |
:rotating_light: *{{ .Rule }}* ({{ .Priority }})
*Pod:* `{{ index .OutputFields "k8s.pod.name" }}` in `{{ index .OutputFields "k8s.ns.name" }}`
*Image:* `{{ index .OutputFields "container.image.repository" }}`
*Command:* `{{ index .OutputFields "proc.cmdline" }}`
*Time:* {{ .Time }}
# PagerDuty: only critical alerts page on-call
pagerduty:
routingKey: "..."
minimumpriority: critical
# Elasticsearch: full audit trail
elasticsearch:
hostport: "http://elasticsearch:9200"
index: falco-alerts
minimumpriority: debug # Store everything
# Prometheus: metrics dashboard
prometheus:
enabled: true
# Exposes: falcosidekick_falco_alerts_total{priority,rule,source,tags,hostname}
# Webhook: custom SIEM or SOAR integration
webhook:
address: "https://your-siem.example.com/api/falco"
customHeaders:
Authorization: "Bearer ${SIEM_API_TOKEN}"
minimumpriority: warning
# OpsGenie: for teams using OpsGenie
opsgenie:
apikey: "..."
minimumpriority: critical
# Teams: Microsoft Teams channel
teams:
webhookurl: "https://outlook.office.com/webhook/..."
minimumpriority: warning
# Grafana annotations: mark security events on dashboards
grafana:
hostport: "http://grafana:3000"
apikey: "..."
dashboardid: 12 # Your main ops dashboard
panelid: 1
minimumpriority: warning
|
Automated response with Falcosidekick + AWS Lambda
1
2
3
4
5
6
7
|
# Trigger a Lambda to quarantine a compromised pod
config:
aws:
lambda:
functionname: "falco-quarantine-pod"
minimumpriority: critical
region: us-east-1
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
|
# lambda_function.py — automatically cordon and isolate compromised pods
import boto3
import json
from kubernetes import client, config
def lambda_handler(event, context):
# Parse Falco alert
alert = json.loads(event['body'])
pod_name = alert['output_fields'].get('k8s.pod.name')
namespace = alert['output_fields'].get('k8s.ns.name')
rule = alert['rule']
if not pod_name or not namespace:
return {'statusCode': 400}
# Only auto-quarantine critical alerts for known attack rules
if alert['priority'] != 'Critical':
return {'statusCode': 200, 'body': 'Non-critical, skipping'}
print(f"QUARANTINE: {pod_name}/{namespace} due to rule: {rule}")
# Load in-cluster config
config.load_incluster_config()
v1 = client.CoreV1Api()
# Label the pod as quarantined
v1.patch_namespaced_pod(
name=pod_name,
namespace=namespace,
body={"metadata": {"labels": {"security.example.com/quarantined": "true"}}}
)
# NetworkPolicy will isolate pods with this label (pre-deployed)
# The pod stays alive for forensics but can't talk to anything
# Notify security team with forensic info
logs = v1.read_namespaced_pod_log(pod_name, namespace, tail_lines=500)
notify_security_team(pod_name, namespace, rule, logs)
return {'statusCode': 200, 'body': f'Quarantined {pod_name}'}
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
# NetworkPolicy: isolate quarantined pods
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: quarantine-isolation
namespace: default
spec:
podSelector:
matchLabels:
security.example.com/quarantined: "true"
policyTypes:
- Ingress
- Egress
# Empty ingress/egress = deny all traffic
|
Kubernetes Audit Log Rules
Beyond syscalls, Falco can analyze the Kubernetes audit log for control-plane level attacks.
Enable K8s audit logging
1
2
3
4
5
6
|
# kube-apiserver configuration
--audit-log-path=/var/log/kubernetes/audit.log
--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-maxage=30
--audit-log-maxbackup=10
--audit-log-maxsize=100
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
# audit-policy.yaml — what to log
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Log all requests to sensitive resources
- level: RequestResponse
resources:
- group: ""
resources: [secrets, serviceaccounts]
- group: rbac.authorization.k8s.io
resources: [clusterroles, clusterrolebindings, roles, rolebindings]
# Log exec/port-forward (common post-exploitation activity)
- level: RequestResponse
resources:
- group: ""
resources: [pods/exec, pods/portforward, pods/attach]
# Log everything else at metadata level
- level: Metadata
|
K8s audit rules for Falco
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
|
# Detect exec into running pods (common investigation AND attack path)
- rule: Exec Pod
desc: >
kubectl exec was used to get a shell in a running pod.
Legitimate for debugging but often used by attackers post-compromise.
condition: >
ka.verb = create
and ka.target.subresource = exec
and not ka.user.name startswith system:
and not ka.user.name in (trusted_kubectl_users)
output: >
Exec into pod
(user=%ka.user.name pod=%ka.target.name ns=%ka.target.namespace
container=%ka.req.pod.containers.name command=%ka.uri.param[command])
priority: NOTICE
source: k8s_audit
tags: [kubernetes, exec, mitre_execution]
# Detect RBAC escalation
- rule: K8s Secret Access
desc: >
A non-system user read a Kubernetes Secret. Could indicate credential
theft or misconfigured RBAC.
condition: >
ka.verb in (get, list, watch)
and ka.target.resource = secrets
and not ka.user.name startswith system:
and ka.response.code = 200
output: >
K8s Secret accessed
(user=%ka.user.name verb=%ka.verb ns=%ka.target.namespace
secret=%ka.target.name response=%ka.response.code)
priority: WARNING
source: k8s_audit
tags: [kubernetes, secrets, mitre_credential_access]
# Anonymous access attempts
- rule: Anonymous Request Allowed
desc: An anonymous (unauthenticated) request was allowed to the K8s API.
condition: >
ka.user.name = system:anonymous
and ka.response.code = 200
output: >
Anonymous request allowed to K8s API
(verb=%ka.verb resource=%ka.target.resource uri=%ka.uri)
priority: ERROR
source: k8s_audit
tags: [kubernetes, authentication, mitre_initial_access]
|
Prometheus Metrics and Grafana Dashboard
Falcosidekick exposes metrics for building a security dashboard:
1
2
3
4
5
6
7
8
9
10
11
|
# Alert rate by priority
sum(rate(falcosidekick_falco_alerts_total[5m])) by (priority)
# Alert rate by rule
topk(10, sum(rate(falcosidekick_falco_alerts_total[1h])) by (rule))
# Critical alerts in last hour (should alert on this)
increase(falcosidekick_falco_alerts_total{priority="Critical"}[1h])
# Alert volume by namespace
sum(rate(falcosidekick_falco_alerts_total[5m])) by (hostname)
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
# Prometheus alert: critical Falco alert received
groups:
- name: falco_security
rules:
- alert: FalcoCriticalAlert
expr: increase(falcosidekick_falco_alerts_total{priority="Critical"}[5m]) > 0
labels:
severity: critical
annotations:
summary: "Falco critical security alert: {{ $labels.rule }}"
runbook_url: "https://runbooks.example.com/falco-critical"
- alert: FalcoHighAlertVolume
expr: sum(rate(falcosidekick_falco_alerts_total[5m])) > 10
for: 5m
labels:
severity: warning
annotations:
summary: "High Falco alert volume ({{ $value }}/s) — possible attack or rule tuning needed"
- alert: FalcoAgentDown
expr: up{job="falco"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Falco agent down on {{ $labels.instance }} — runtime security is blind"
|
Falco in CI/CD: Detecting Malicious Build Behavior
Run Falco during CI builds to detect supply-chain attacks (a compromised dependency running code during npm install):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
|
# GitHub Actions: run Falco alongside the build
jobs:
secure-build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Start Falco
run: |
# Install Falco in syscall capture mode (no K8s)
curl -fsSL https://falco.org/install.sh | bash
sudo falco \
--modern-bpf \
--json-output \
--rules-file /etc/falco/falco_rules.yaml \
--rules-file ./ci-falco-rules.yaml \
> /tmp/falco-output.jsonl &
echo "FALCO_PID=$!" >> $GITHUB_ENV
- name: Build (under Falco observation)
run: |
npm ci
npm run build
npm test
- name: Check Falco alerts
if: always()
run: |
kill $FALCO_PID || true
# Fail the build if any WARNING+ alerts fired during build
CRITICAL=$(cat /tmp/falco-output.jsonl | \
jq 'select(.priority == "Critical" or .priority == "Error" or .priority == "Warning")' | \
wc -l)
if [ "$CRITICAL" -gt 0 ]; then
echo "::error::Falco detected suspicious activity during build!"
cat /tmp/falco-output.jsonl | jq 'select(.priority != "Debug" and .priority != "Informational")'
exit 1
fi
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
# ci-falco-rules.yaml — rules specific to build environments
- rule: Build Process Network Exfiltration
desc: >
Build process made an unexpected network connection. Could indicate
a compromised dependency exfiltrating code or secrets.
condition: >
outbound
and not proc.name in (npm, node, pip, python, curl, wget, git, go, cargo, mvn, gradle)
and not fd.sport in (80, 443, 22)
output: >
Unexpected network connection during build
(proc=%proc.name cmdline=%proc.cmdline connection=%fd.name)
priority: WARNING
|
Falco’s syscall interception has a measurable but manageable overhead:
| Driver |
Typical CPU overhead |
Notes |
| Kernel module |
1–3% |
Fastest interception path |
| Legacy eBPF |
2–4% |
Safer, slightly more overhead |
| Modern eBPF (CO-RE) |
2–5% |
Best compatibility, recommended |
To reduce overhead:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
# falco.yaml — performance tuning
# Increase syscall buffer size to reduce drops under high load
syscall_buf_size_preset: 4 # Range 1-10, default 4. Higher = more memory, fewer drops
# Drop syscalls not needed by any rule (Falco does this automatically based on loaded rules)
# Explicitly set the set of interesting syscalls if you know you don't need certain ones:
base_syscalls:
custom_set: [execve, execveat, open, openat, connect, accept, ptrace, setuid, setgid, mount]
repair: true
# Monitor for syscall drops
metrics:
enabled: true
interval: 15m
output_rule: true
resource_utilization_enabled: true
|
1
2
3
4
5
6
|
# Check if Falco is dropping events (means host is too busy for the buffer)
kubectl logs -n falco daemonset/falco | grep "Falco internal: syscall event drop"
# If drops are occurring, increase buffer:
helm upgrade falco falcosecurity/falco --reuse-values \
--set "falco.syscall_buf_size_preset=6"
|
Quick Reference: Default Rules Worth Knowing
Falco ships with a comprehensive default ruleset. The highest-value rules to understand:
| Rule |
What it catches |
Terminal shell in container |
Interactive shell spawned in a running container |
Write below binary dir |
Writes to /bin, /sbin, /usr/bin etc. |
Write below etc |
Writes to /etc in a container |
Read sensitive file untrusted |
Reads of /etc/shadow, /etc/sudoers, SSH keys |
Run shell untrusted |
Unexpected shell from non-shell parent process |
Outbound or Inbound Traffic not to Authorized Server Process |
Unexpected network activity |
System user interactive |
System accounts (www-data, nobody) running interactive sessions |
Modify binary dirs |
Modification of system binary directories |
Change thread namespace |
setns calls — namespace escape attempts |
Launch Privileged Container |
docker run --privileged or equivalent |
Contact K8S API Server From Container |
Pod calling K8s API directly |
Packet socket created in container |
Raw socket creation — could be network sniffing |
Summary
Falco gives you visibility that no other tool provides: ground truth about what is actually happening inside your running containers at the kernel level. No container runtime, no language, no obfuscation technique can hide syscalls from a properly deployed Falco instance.
The deployment path:
- Install with modern eBPF driver — no kernel module required, CO-RE means it survives kernel upgrades
- Start with default rules in NOTICE mode — observe the noise before tuning
- Audit what fires — identify legitimate processes triggering alerts, add exceptions
- Write custom rules for your threat model — what would an attacker do in your environment?
- Route alerts with Falcosidekick — Slack for awareness, PagerDuty for critical, Elasticsearch for retention
- Build automated response — NetworkPolicy quarantine for Critical alerts, Lambda/webhook for remediation
- Add K8s audit rules — cover the control plane, not just the data plane
- Integrate into CI/CD — catch supply-chain attacks during builds
The goal isn’t zero alerts — it’s a tuned ruleset where every alert represents a genuine investigation decision. When a Falco alert fires in production and you know it means something real happened, you have exactly what a security program needs: reliable signal in the noise.
Comments