Chaos Engineering in Practice: Breaking Things on Purpose to Build Unbreakable Systems

March 27, 2026 21 min read DevOps

srechaos-engineeringreliabilitykuberneteslitmusresiliencetesting

Contents

Chaos Engineering in Practice: Breaking Things on Purpose to Build Unbreakable Systems

In 2011, Netflix engineers had a problem: they were moving to AWS and needed confidence that their system would survive infrastructure failures. They couldn’t just hope it would work — they needed to prove it. So they built Chaos Monkey, a tool that randomly terminated EC2 instances in production during business hours. If a random instance could die at any moment, engineers were forced to build systems that could handle it gracefully.

This is the core premise of chaos engineering: controlled experiments that inject failures into a system to discover weaknesses before they become incidents. Not random destruction — disciplined, hypothesis-driven experiments with defined blast radius and rollback plans.

The discipline has matured enormously since Chaos Monkey. Today, chaos engineering covers network partition simulation, latency injection, resource exhaustion, dependency failures, and even cognitive experiments (game days, fire drills). This guide walks through the full practice: the principles, the toolchain, writing good experiments, running game days, and building chaos into your delivery pipeline.

The Principles of Chaos Engineering

The Principles of Chaos Engineering define four core ideas:

1. Build a hypothesis around steady-state behavior

Before you can detect a deviation, you need to define normal. Steady-state is the measurable, observable behavior of a healthy system:

P99 API latency < 200ms
Order completion rate > 99.9%
Error rate < 0.1%
Active websocket connections: 10,000–15,000

The hypothesis: “This steady state will continue to hold even when [failure condition X] is introduced.”

If it doesn’t hold, you’ve found a real weakness before your users did.

2. Vary real-world events

Failures worth simulating are failures that actually happen:

Instances crash (hardware failure, OOM killer, deployment gone wrong)
Network packets are delayed or dropped (cloud provider issues, noisy neighbors)
Dependencies return errors or time out (upstream API degradation)
Disks fill up (runaway logging, lack of retention policy)
CPU is saturated (traffic spike, runaway process)
DNS resolution fails (misconfigured resolver, provider outage)

Simulating unrealistic failures produces unrealistic results.

3. Run experiments in production

This is the uncomfortable truth: production has state, traffic patterns, data volumes, and third-party integrations that staging never fully replicates. A circuit breaker that opens correctly in staging may never trigger in production because the thresholds were tuned against synthetic load.

Start with the lowest-risk experiments in staging. Graduate to production as confidence grows. The goal is always production — that’s where the real risk lives.

4. Minimize blast radius

Every experiment needs a defined blast radius — the maximum scope of impact if something goes wrong. Start small:

One pod → one deployment → one service → one region
1% of traffic → 5% → 10%
30-second experiment → 5-minute → steady-state injection

Automate rollback: if your reliability metrics drop below a threshold, the experiment stops automatically.

The Experiment Design Process

A well-designed chaos experiment follows a structured template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


## Experiment: Payment Service Database Connection Pool Exhaustion

### Hypothesis
The payment service will continue processing orders (with degraded throughput)
and return graceful errors to callers even when the database connection pool
is 90% saturated by slow queries.

### Steady State
- payment_service_order_completion_rate > 99.5%
- payment_service_p99_latency < 500ms
- payment_service_error_rate < 0.1%
- payment_service_circuit_breaker_state = closed

### Failure to Inject
- Inject artificial 5-second delay on 20% of DB queries in the payment service
- Duration: 10 minutes
- Blast radius: payment-service pods only; does not affect DB itself

### Rollback Triggers
- Stop experiment immediately if:
  - error_rate > 5%
  - Any payment processing system alert fires
  - On-call engineer requests stop

### Rollback Procedure
1. Remove the latency injection (automated via Litmus abort)
2. Verify steady state returns within 2 minutes
3. Page on-call if steady state doesn't recover

### Observations to Make
- Does the circuit breaker open? At what threshold?
- Do queued requests drain after injection stops?
- Are callers informed with appropriate error codes (503 vs 500)?
- Are DB connection timeouts logged with actionable context?

### Expected Outcome
Circuit breaker opens at ~80% pool saturation, returning 503 to callers.
Pool drains within 60 seconds after injection stops.

### Actual Outcome
[Filled in after experiment]

Toolchain Overview

Tool	Best for	Language
Chaos Monkey	Random EC2/instance termination	JVM
Litmus Chaos	Kubernetes-native, GitOps-friendly	YAML/Go
Chaos Mesh	Kubernetes, fine-grained network faults	YAML/Go
Gremlin	Managed SaaS, enterprise features	Cloud
k6	Load testing + latency injection	JavaScript
Pumba	Docker container chaos	Go
tc/netem	Raw Linux network fault injection	Shell
toxiproxy	Proxy-based latency/error injection	Go

For most teams, Litmus Chaos (Kubernetes-native) + k6 (load + chaos) + toxiproxy (dependency simulation) covers the vast majority of scenarios.

Litmus Chaos: Kubernetes-Native Chaos Engineering

Litmus is a CNCF incubating project that runs chaos experiments as Kubernetes CRDs. Experiments are declarative, version-controlled, and integrate naturally with GitOps.

Installation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm
helm repo update

helm install litmus litmuschaos/litmus \
  --namespace litmus \
  --create-namespace \
  --set portal.frontend.service.type=ClusterIP \
  --set mongodb.enabled=true

# Install the chaos experiment catalog
kubectl apply -f https://hub.litmuschaos.io/api/chaos/master?file=charts/generic/experiments.yaml

Core concepts

ChaosEngine        → Runs one or more experiments against a target
ChaosExperiment    → Defines what chaos to inject (pod-delete, network-loss, etc.)
ChaosResult        → Stores results and pass/fail verdict
ServiceAccount     → RBAC for the chaos runner

Experiment 1: Pod deletion (the classic)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72


# chaos-engine-pod-delete.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: payment-service-pod-delete
  namespace: payments
spec:
  appinfo:
    appns: payments
    applabel: "app=payment-service"
    appkind: deployment

  # Stop if steady state checks fail
  jobCleanUpPolicy: delete

  # Monitoring integration
  monitoring: true
  annotationCheck: "false"

  chaosServiceAccount: litmus-admin

  experiments:
    - name: pod-delete
      spec:
        probe:
          # Steady-state probe: check error rate before AND after
          - name: check-payment-error-rate
            type: promProbe
            promProbe/inputs:
              endpoint: "http://prometheus:9090"
              query: |
                sum(rate(http_requests_total{service="payment-service",status=~"5.."}[2m]))
                /
                sum(rate(http_requests_total{service="payment-service"}[2m]))
              comparator:
                type: float
                criteria: "<="
                value: "0.01"  # Error rate must stay below 1%
            mode: Continuous
            runProperties:
              probeTimeout: 10
              interval: 15
              retry: 2
              probePollingInterval: 2

          # HTTP probe: payment endpoint stays responsive
          - name: check-payment-endpoint
            type: httpProbe
            httpProbe/inputs:
              url: "http://payment-service.payments.svc.cluster.local/healthz"
              insecureSkipVerify: false
              responseTimeout: 3000
              method:
                get:
                  criteria: ==
                  responseCode: "200"
            mode: Continuous
            runProperties:
              probeTimeout: 10
              interval: 5
              retry: 3

        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "300"  # 5 minutes
            - name: CHAOS_INTERVAL
              value: "30"   # Delete a pod every 30 seconds
            - name: FORCE
              value: "false"  # Graceful termination
            - name: PODS_AFFECTED_PERC
              value: "50"  # Kill up to 50% of pods at a time

Experiment 2: Network latency injection

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: payment-service-network-latency
  namespace: payments
spec:
  appinfo:
    appns: payments
    applabel: "app=payment-service"
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-network-latency
      spec:
        probe:
          - name: check-p99-latency
            type: promProbe
            promProbe/inputs:
              endpoint: "http://prometheus:9090"
              query: |
                histogram_quantile(0.99,
                  rate(http_request_duration_seconds_bucket{service="payment-service"}[2m])
                )
              comparator:
                type: float
                criteria: "<="
                value: "2.0"  # P99 must stay below 2 seconds
            mode: Continuous
            runProperties:
              probeTimeout: 10
              interval: 15
              retry: 2
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "300"
            - name: NETWORK_LATENCY
              value: "2000"   # 2000ms added latency
            - name: JITTER
              value: "200"    # ±200ms jitter
            - name: TARGET_PODS
              value: ""       # Empty = all matching pods
            - name: PODS_AFFECTED_PERC
              value: "50"     # Affect 50% of pods
            - name: CONTAINER_RUNTIME
              value: "containerd"
            - name: SOCKET_PATH
              value: "/run/containerd/containerd.sock"

Experiment 3: CPU stress

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: frontend-cpu-stress
  namespace: production
spec:
  appinfo:
    appns: production
    applabel: "app=frontend"
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-cpu-hog
      spec:
        probe:
          - name: check-frontend-availability
            type: httpProbe
            httpProbe/inputs:
              url: "https://app.example.com/healthz"
              responseTimeout: 5000
              method:
                get:
                  criteria: ==
                  responseCode: "200"
            mode: Continuous
            runProperties:
              probeTimeout: 10
              interval: 10
              retry: 3
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "180"
            - name: CPU_CORES
              value: "2"      # Consume 2 CPU cores
            - name: CPU_LOAD
              value: "80"     # 80% utilization per core
            - name: PODS_AFFECTED_PERC
              value: "100"    # All frontend pods (tests autoscaling)

Experiment 4: Disk I/O fill

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: logging-disk-fill
  namespace: production
spec:
  appinfo:
    appns: production
    applabel: "app=log-processor"
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
    - name: disk-fill
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "120"
            - name: FILL_PERCENTAGE
              value: "80"     # Fill ephemeral storage to 80%
            - name: DATA_BLOCK_SIZE
              value: "256"    # 256KB blocks

ChaosSchedule: recurring experiments

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


# Run pod-delete every weekday at 11am — always be ready
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosSchedule
metadata:
  name: payment-weekly-chaos
  namespace: payments
spec:
  schedule:
    # Cron expression: weekdays at 11:00 AM UTC
    cron: "0 11 * * 1-5"
  engineTemplateSpec:
    appinfo:
      appns: payments
      applabel: "app=payment-service"
      appkind: deployment
    chaosServiceAccount: litmus-admin
    experiments:
      - name: pod-delete
        spec:
          components:
            env:
              - name: TOTAL_CHAOS_DURATION
                value: "120"
              - name: CHAOS_INTERVAL
                value: "30"
              - name: PODS_AFFECTED_PERC
                value: "33"

Toxiproxy: Simulating Dependency Failures

Toxiproxy is a proxy that sits between your service and its dependencies, injecting controlled faults. It’s perfect for simulating upstream API degradation without touching infrastructure.

Setup

1
2
3
4
5
6
7


# Run toxiproxy in Kubernetes as a sidecar or dedicated deployment
docker run -d \
  --name toxiproxy \
  -p 8474:8474 \
  -p 5432:5432 \   # Proxy port for PostgreSQL
  -p 6379:6379 \   # Proxy port for Redis
  ghcr.io/shopify/toxiproxy:latest

1
2
3


# Configure via CLI or API
toxiproxy-cli create postgres-proxy --listen 0.0.0.0:5432 --upstream postgres:5432
toxiproxy-cli create redis-proxy --listen 0.0.0.0:6379 --upstream redis:6379

Experiment recipes with toxiproxy

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# Inject 500ms latency on all database connections
toxiproxy-cli toxic add postgres-proxy \
  --type latency \
  --attribute latency=500 \
  --attribute jitter=100 \
  --toxicName db-latency

# Cause 20% of database connections to time out
toxiproxy-cli toxic add postgres-proxy \
  --type timeout \
  --attribute timeout=30000 \
  --toxicName db-timeout \
  --downstream

# Drop 10% of packets to Redis (simulates network degradation)
toxiproxy-cli toxic add redis-proxy \
  --type slow_close \
  --attribute delay=5000 \
  --toxicName redis-slow-close

# Limit bandwidth to 56kbps (simulates congested uplink)
toxiproxy-cli toxic add redis-proxy \
  --type bandwidth \
  --attribute rate=7 \  # 7 KB/s = ~56kbps
  --toxicName redis-bandwidth

# Remove a toxic when done
toxiproxy-cli toxic remove postgres-proxy --toxicName db-latency

Toxiproxy in integration tests

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


// Go — toxiproxy in automated tests
package integration_test

import (
    "testing"
    "time"
    toxiproxy "github.com/shopify/toxiproxy/v2/client"
)

func TestPaymentServiceDatabaseLatency(t *testing.T) {
    client := toxiproxy.NewClient("http://toxiproxy:8474")
    proxy, _ := client.Proxy("postgres-proxy")

    // Inject 800ms latency — above our circuit breaker threshold
    toxic, err := proxy.AddToxic("db-slow", "latency", "downstream", 1.0,
        toxiproxy.Attributes{"latency": 800, "jitter": 50},
    )
    if err != nil {
        t.Fatalf("Failed to add toxic: %v", err)
    }
    defer proxy.RemoveToxic(toxic.Name)

    // Give the system a moment to feel the latency
    time.Sleep(2 * time.Second)

    // Assert circuit breaker opened
    resp, _ := http.Get("http://payment-service/metrics")
    metrics := parseMetrics(resp.Body)
    if metrics["circuit_breaker_state"] != "open" {
        t.Error("Expected circuit breaker to open under database latency")
    }

    // Remove toxic and verify recovery
    proxy.RemoveToxic(toxic.Name)
    time.Sleep(10 * time.Second)

    if metrics["circuit_breaker_state"] != "closed" {
        t.Error("Expected circuit breaker to recover after latency removed")
    }
}

k6: Load Testing + Chaos

k6 is primarily a load testing tool, but its extension ecosystem and scripting flexibility make it powerful for chaos-adjacent scenarios: traffic spike simulation, stress testing under degraded conditions, and validating graceful degradation.

Installation

1
2
3


brew install k6
# or
docker pull grafana/k6

Basic load test with chaos validation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73


// load-test-with-chaos.js
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Rate, Trend } from 'k6/metrics';

// Custom metrics to track during chaos
const errorRate = new Rate('payment_errors');
const checkoutDuration = new Trend('checkout_duration_ms');

export const options = {
  stages: [
    { duration: '2m', target: 50 },   // Ramp to 50 users
    { duration: '5m', target: 50 },   // Steady state (chaos injects here)
    { duration: '2m', target: 200 },  // Spike to 200 users
    { duration: '5m', target: 200 },  // Hold spike
    { duration: '2m', target: 0 },    // Ramp down
  ],
  thresholds: {
    // SLO: 99% of requests complete successfully
    'payment_errors': ['rate<0.01'],
    // SLO: P95 checkout under 1 second
    'checkout_duration_ms': ['p(95)<1000'],
    // SLO: P99 checkout under 2 seconds
    'checkout_duration_ms': ['p(99)<2000'],
    // Overall HTTP error rate
    'http_req_failed': ['rate<0.01'],
  },
};

const BASE_URL = 'https://api.example.com';

export default function () {
  // Simulate a user checkout flow
  const startTime = Date.now();

  // 1. Add item to cart
  const cartResp = http.post(`${BASE_URL}/api/cart`, JSON.stringify({
    product_id: 'prod_123',
    quantity: 1,
  }), { headers: { 'Content-Type': 'application/json' } });

  check(cartResp, {
    'cart: status 200': (r) => r.status === 200,
  });

  sleep(1);

  // 2. Checkout
  const checkoutResp = http.post(`${BASE_URL}/api/checkout`, JSON.stringify({
    cart_id: cartResp.json('cart_id'),
    payment_method: 'card_test',
  }), { headers: { 'Content-Type': 'application/json' } });

  const checkoutTime = Date.now() - startTime;
  checkoutDuration.add(checkoutTime);

  const success = check(checkoutResp, {
    'checkout: status 200': (r) => r.status === 200,
    'checkout: has order_id': (r) => r.json('order_id') !== undefined,
  });

  errorRate.add(!success);

  sleep(Math.random() * 3 + 1);
}

// Summary output
export function handleSummary(data) {
  return {
    'results/load-test-summary.json': JSON.stringify(data, null, 2),
    stdout: textSummary(data, { indent: ' ', enableColors: true }),
  };
}

Running k6 against Kubernetes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


# k6-test-job.yaml — run k6 as a Kubernetes Job
apiVersion: batch/v1
kind: Job
metadata:
  name: chaos-load-test
  namespace: testing
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: k6
          image: grafana/k6:latest
          command: ["k6", "run", "--out", "influxdb=http://influxdb:8086/k6", "/scripts/test.js"]
          volumeMounts:
            - name: test-scripts
              mountPath: /scripts
          env:
            - name: K6_PROMETHEUS_RW_SERVER_URL
              value: "http://prometheus:9090/api/v1/write"
      volumes:
        - name: test-scripts
          configMap:
            name: k6-scripts

Combining k6 with Litmus

Run load test and chaos experiment simultaneously to see how the system behaves under combined stress:

1
2
3
4
5
6
7
8


# Terminal 1: Run load test
k6 run --out prometheus=http://prometheus:9090/api/v1/write load-test.js &

# Terminal 2: Inject chaos while load is running
kubectl apply -f chaos-engine-pod-delete.yaml

# Terminal 3: Watch the metrics dashboard
# Grafana: watch error rate and latency as pods are deleted under load

Game Days: Structured Chaos at Human Scale

A game day is a scheduled event where a team simulates a realistic failure scenario in a controlled setting. Unlike automated chaos experiments that test technical resilience, game days test human resilience: on-call procedures, communication, runbooks, escalation paths.

Game day structure

Duration: 2–4 hours
Participants: On-call engineers, incident commander, observers

PHASE 1: Preparation (30 min before)
  - Brief all participants on the scenario and their roles
  - Ensure monitoring dashboards are up
  - Designate a "chaos controller" who can stop the experiment
  - Set up a dedicated Slack channel for the exercise

PHASE 2: Scenario injection (5 min)
  - Chaos controller secretly applies the failure
  - Participants don't know exactly what failed or when

PHASE 3: Detection and response (60–90 min)
  - Participants respond as if this were a real incident
  - Incident commander runs the call
  - Follow your actual runbooks and escalation procedures

PHASE 4: Resolution
  - Chaos controller removes the failure
  - Team verifies recovery

PHASE 5: Retrospective (45 min)
  - What was detected, and how quickly?
  - What wasn't detected?
  - Which runbooks were useful? Which were wrong or missing?
  - What would have reduced MTTR?
  - Action items with owners and due dates

Game day scenario library

Scenario 1: Database primary failover

Inject: Kill the primary database pod
Tests: Does automatic failover work? How long does it take? Do applications handle the brief outage gracefully? Do runbooks cover this?

Scenario 2: Certificate expiry

Inject: Point a service at an expired TLS certificate
Tests: Do alerts fire before certificates actually expire? Is there a runbook? Who owns certificate rotation?

Scenario 3: Cascading dependency failure

Inject: Make the auth service return 503 for all requests
Tests: Do all dependent services degrade gracefully? Do circuit breakers open? Is the blast radius contained to auth-dependent flows?

Scenario 4: Region outage simulation

Inject: Cordon all nodes in one availability zone
Tests: Does traffic shift to other AZs? Is there enough capacity? Do PodDisruptionBudgets prevent full outage?

Scenario 5: Silent data corruption

Inject: Have a service return subtly wrong data (e.g., off-by-one prices)
Tests: Do we have data validation? Are there sanity checks in downstream consumers? Would users notice before engineers did?

Scenario 6: Alert storm

Inject: Trigger dozens of simultaneous alerts
Tests: Can the on-call engineer identify the root cause in an alert storm? Are alerts correlated? Is runbook navigation fast?

Game day runbook template

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


# Game Day: Database Failover — 2026-04-15

## Scenario
The primary PostgreSQL instance becomes unavailable. Replicas are healthy.

## Success Criteria
- [ ] Failover detected within 5 minutes via alerting
- [ ] Application traffic recovered within 10 minutes
- [ ] No data loss
- [ ] Incident correctly escalated per runbook

## Chaos Controller Actions
1. At T+0: `kubectl delete pod postgres-primary-0 -n databases --force`
2. At T+45min (or upon request): Remove chaos, verify recovery
3. Do NOT reveal the scenario until the retrospective

## Observer Notes
- Time of first alert: ___
- Time on-call acknowledged: ___
- Time root cause identified: ___
- Time traffic recovered: ___
- Runbooks consulted: ___
- Actions taken that weren't in runbooks: ___

## Retrospective Notes
[Filled in after the game day]

Measuring the Impact of Chaos Engineering

Chaos engineering should produce measurable improvements. Track these metrics over time:

Mean Time to Detection (MTTD)

How long from failure injection to first alert? If your MTTD is 15 minutes, an attacker or silent failure has a 15-minute undetected window.

Mean Time to Recovery (MTTR)

From failure to restored service. Compare MTTR during game days vs real incidents — the gap reveals how much your runbooks and tooling help.

Failure coverage

What percentage of your failure modes have been tested? Build a matrix:

1
2
3
4
5
6
7
8


| Failure Mode            | Tested? | Last Run  | Outcome  |
|-------------------------|---------|-----------|----------|
| Pod deletion            | ✅      | 2026-03-15 | Passed  |
| Database latency        | ✅      | 2026-03-01 | Failed — circuit breaker didn't open |
| Redis connection loss   | ✅      | 2026-02-20 | Passed  |
| CPU saturation          | ❌      | Never     | —       |
| Certificate expiry      | ❌      | Never     | —       |
| Zone failure            | ✅      | 2026-01-10 | Passed  |

Chaos-driven improvements

Track action items from experiments and game days:

Bugs filed and resolved
Runbooks created or updated
Alerts added
Architecture changes made
MTTR improvement from before/after

Building a Chaos Program: The Maturity Model

Level 0: No chaos engineering

You discover failures when users report them
MTTR measured in hours
Runbooks are incomplete or nonexistent

Level 1: Manual experiments in staging

Ad-hoc chaos runs by interested engineers
No steady-state definition
Results aren’t tracked systematically

Level 2: Structured experiments in staging

Experiments follow the hypothesis template
Steady state is defined and monitored
Results are documented
Game days happen quarterly

Level 3: Automated chaos in production (limited)

Litmus/Chaos Mesh running scheduled experiments
Automated steady-state probes halt experiments on violation
Monthly game days
Chaos results feed back into architecture decisions

Level 4: Continuous chaos in production

Chaos experiments run continuously (not just scheduled)
Metrics from chaos experiments are in the same dashboard as SLOs
Engineers are notified of new failure modes during code review
Game days are monthly and cover multi-system scenarios
MTTR measured in minutes

Most teams should aim for Level 3. Level 4 is Netflix-scale and requires significant investment in tooling and culture.

Integrating Chaos into CI/CD

Automated chaos gates in your delivery pipeline catch regressions before they reach production:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53


# .github/workflows/chaos-validation.yaml
name: Chaos Validation

on:
  push:
    branches: [main]

jobs:
  deploy-staging:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to staging
        run: |
          helm upgrade --install payment-service ./charts/payment-service \
            --namespace staging \
            --values values-staging.yaml

  chaos-validation:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - name: Wait for deployment to stabilize
        run: sleep 60

      - name: Run chaos experiment
        run: |
          kubectl apply -f ci/chaos-engine-pod-delete.yaml
          # Wait for experiment to complete
          kubectl wait --for=condition=completed \
            chaosengine/payment-chaos-ci \
            --namespace staging \
            --timeout=600s

      - name: Check chaos result
        run: |
          VERDICT=$(kubectl get chaosresult payment-chaos-ci-pod-delete \
            -n staging \
            -o jsonpath='{.status.experimentStatus.verdict}')

          echo "Chaos experiment verdict: $VERDICT"

          if [ "$VERDICT" != "Pass" ]; then
            echo "::error::Chaos experiment failed — service is not resilient to pod deletion"
            kubectl describe chaosresult payment-chaos-ci-pod-delete -n staging
            exit 1
          fi

      - name: Promote to production (only if chaos passed)
        run: |
          helm upgrade --install payment-service ./charts/payment-service \
            --namespace production \
            --values values-production.yaml

Common Mistakes and How to Avoid Them

Running chaos without observability If you can’t see what’s happening, you can’t learn from the experiment. Before injecting any fault, verify your metrics, traces, and logs are flowing. If Prometheus is down, the chaos experiment teaches you nothing.

No defined steady state “It felt fine” is not a steady-state measurement. Define concrete, measurable thresholds before the experiment. Otherwise you can’t tell whether the experiment passed or failed.

Skipping staging Even if the goal is production chaos, start every new experiment type in staging first. Validate that the experiment mechanism works, that steady-state probes function, and that rollback works correctly.

Too large a blast radius too fast Kill one pod, not all pods. Inject latency on 10% of requests, not 100%. You want to find weaknesses, not create actual incidents.

No stakeholder communication Inform your team before running chaos. If an engineer sees a flapping alert and doesn’t know a chaos experiment is running, they’ll burn time debugging something intentional. Add a calendar event and Slack message before every experiment.

Running during risky windows Don’t run chaos experiments on Fridays, during deployments, during peak traffic hours, or within 48 hours of a major release. Pick boring Tuesday mornings at 30% traffic.

Ignoring the results The experiment that reveals a weakness is the most valuable result you can get — and also the most likely to be ignored. Every failed experiment should produce an issue in your tracker and an owner within 24 hours.

Quick Reference: Litmus ChaosHub Experiments

Experiment	What it injects	Key parameters
`pod-delete`	Kills pods randomly	`CHAOS_INTERVAL`, `PODS_AFFECTED_PERC`
`pod-network-latency`	Adds latency to pod network	`NETWORK_LATENCY`, `JITTER`
`pod-network-loss`	Drops packets	`NETWORK_PACKET_LOSS_PERCENTAGE`
`pod-network-corruption`	Corrupts packets	`NETWORK_PACKET_CORRUPTION_PERCENTAGE`
`pod-cpu-hog`	Saturates CPU	`CPU_CORES`, `CPU_LOAD`
`pod-memory-hog`	Exhausts memory	`MEMORY_CONSUMPTION`
`disk-fill`	Fills ephemeral storage	`FILL_PERCENTAGE`
`node-drain`	Drains a Kubernetes node	`TARGET_NODE`
`node-taint`	Taints a node	`TAINT_LABEL`
`pod-io-stress`	Saturates disk I/O	`FILESYSTEM_UTILIZATION_PERCENTAGE`
`container-kill`	Kills a specific container	`TARGET_CONTAINER`

Summary

Chaos engineering is how you convert “I think our system is reliable” into “I have evidence our system is reliable.” The discipline builds confidence through deliberate failure:

Define steady state — concrete, measurable metrics that describe a healthy system
Hypothesize — write a specific, falsifiable prediction about system behavior under failure
Start small — one pod, short duration, automated rollback
Instrument first — don’t inject chaos you can’t observe
Graduate to production — staging gives you mechanics; production gives you truth
Hold game days — human resilience is as important as technical resilience
Track improvements — chaos that doesn’t produce action items is just destruction

The teams that practice chaos engineering consistently discover that their systems are less reliable than they believed — and then they fix it. The teams that don’t practice it discover the same thing, except their users find out first.

Chaos Engineering in Practice: Breaking Things on Purpose to Build Unbreakable Systems

The Principles of Chaos Engineering

1. Build a hypothesis around steady-state behavior

2. Vary real-world events

3. Run experiments in production

4. Minimize blast radius

The Experiment Design Process

Toolchain Overview

Litmus Chaos: Kubernetes-Native Chaos Engineering

Installation

Core concepts

Experiment 1: Pod deletion (the classic)

Experiment 2: Network latency injection

Experiment 3: CPU stress

Experiment 4: Disk I/O fill

ChaosSchedule: recurring experiments

Toxiproxy: Simulating Dependency Failures

Setup

Experiment recipes with toxiproxy

Toxiproxy in integration tests

k6: Load Testing + Chaos

Installation

Basic load test with chaos validation

Running k6 against Kubernetes

Combining k6 with Litmus

Game Days: Structured Chaos at Human Scale

Game day structure

Game day scenario library

Game day runbook template

Measuring the Impact of Chaos Engineering

Mean Time to Detection (MTTD)

Mean Time to Recovery (MTTR)

Failure coverage

Chaos-driven improvements

Building a Chaos Program: The Maturity Model

Level 0: No chaos engineering

Level 1: Manual experiments in staging

Level 2: Structured experiments in staging

Level 3: Automated chaos in production (limited)

Level 4: Continuous chaos in production

Integrating Chaos into CI/CD

Common Mistakes and How to Avoid Them

Quick Reference: Litmus ChaosHub Experiments

Summary

Comments