A backup you haven’t tested is not a backup. It’s a hope.

That distinction matters enormously at 2am when your primary database is gone, your storage array is throwing errors you’ve never seen before, and you’re trying to remember if you ever actually verified that the weekly backup job ran successfully. Most people have backups configured. Very few have disaster recovery.

Backups are an input. Disaster recovery is the process — the documented, tested, rehearsed sequence of steps that gets your systems back to a known-good state within a time window your organization can survive. This guide covers what that actually means: how to define your recovery targets, classify your systems, build real runbooks, inject controlled failures through chaos engineering, and most importantly, test your restores before you need them.

The Uncomfortable Truth About Outages

Let’s talk about what an outage actually costs before we get into the mechanics.

For a business, the math is direct: revenue per hour multiplied by duration, plus staff time chasing the incident, plus the longer-tail cost of customer trust lost. For a homelab or self-hosted setup, the math is different but still real: your weekend, your family’s patience with “I just need to fix one more thing,” the media server that’s down while guests are over, the Home Assistant automations that stopped working three days before anyone noticed.

Real disaster scenarios that happen to real people:

• Ransomware: A single phishing email or unpatched vulnerability encrypts everything attached to your network, including your NAS and any backup destination mounted as a network share
• Hardware failure: A drive dies in a RAID array that was already degraded, or a controller card fails and takes all drives with it
• Accidental deletion: rm -rf /data aimed at the wrong directory, a DROP TABLE run against production instead of staging, a misconfigured Ansible playbook that wipes configs
• Datacenter/provider outage: Your VPS provider goes dark for 6 hours, or your home loses power for 3 days after a storm
• Filesystem corruption: Power loss during a write, a kernel bug, or a hardware glitch leaves your ext4 or ZFS pool unclean
• Bad deployment: A code change or config update makes your service fail in a way that corrupts data before anyone notices

The difference between a disaster and an incident is whether you have a plan. An incident is something you solve. A disaster is something that happens to you while you scramble.

RTO and RPO: The Two Numbers That Define Your Strategy

Every conversation about disaster recovery eventually arrives at two acronyms. They look similar but measure fundamentally different things.

RPO — Recovery Point Objective is the maximum amount of data loss you can tolerate, expressed as time. If your RPO is 1 hour, you can afford to lose up to 1 hour of data. This directly drives your backup frequency: your most recent backup must be no older than your RPO.

RTO — Recovery Time Objective is the maximum amount of time your system can be unavailable before the impact becomes unacceptable. If your RTO is 4 hours, you need to be fully operational within 4 hours of declaring a disaster.

These numbers are not technical decisions. They are business decisions that have technical and financial consequences.

The Cost Curve

Achieving lower RPO and RTO costs money — not linearly, but exponentially:

The combination of these targets drives your entire architecture. A RPO of 0 and RTO of 5 minutes for every system would be prohibitively expensive for almost any organization. The discipline is being honest about which systems actually need which targets.

System Tiers: Matching Recovery Targets to Criticality

Not all systems are equal. Classify them:

There’s an uncomfortable irony in Tier 0: it includes DNS, authentication, and monitoring — the very things you need to recover everything else. If your monitoring system goes down in a disaster, you lose visibility. If your auth server is gone, you can’t log into other systems. These need to be recovered first and require special handling.

Applying this to a homelab:

• Tier 0: Pi-hole/AdGuard (DNS), Authelia/authentik (auth), Uptime Kuma (monitoring)
• Tier 1: Home Assistant, primary file storage, media server database
• Tier 2: Git server, password manager sync, personal wiki
• Tier 3: Dev VMs, experimental containers
• Tier 4: Build artifacts, downloaded ISOs, cached data

Classifying Your Systems and Mapping Dependencies

Before you can recover, you need to know what you have. Create a service inventory. For each service, document:

• Name and description
• Where it runs (host, VM, container, cloud)
• Data it owns (databases, volumes, config files)
• Upstream dependencies (what it needs to function)
• Downstream dependencies (what breaks if this goes down)
• Tier classification
• Backup location and schedule
• Responsible owner

Keep this in a simple markdown table or spreadsheet. Keep it in git. The only inventory that matters is one that’s current.

Dependency mapping matters more than people realize. A service failure cascades through dependencies in ways that aren’t obvious until they happen. If your Redis cache goes down and your application doesn’t handle cache misses gracefully, your app is effectively down even though nothing touched it directly. If your Vault server is unavailable and your apps fetch secrets at startup, restarting a container causes it to fail even though the original problem had nothing to do with it.

Draw the dependency graph. For each system, ask: “If this goes down right now, what else breaks within 5 minutes? Within an hour?”

Backup Strategies by Data Type

Different data types have different backup tools. Using the right tool matters.

Databases

PostgreSQL

For small to medium databases, pg_dump is the baseline. It produces a logical dump — SQL statements that can restore the database on any Postgres version:

1
2
3
4
5
6
7
8

# Dump a single database
pg_dump -h localhost -U postgres -d myapp -F c -f /backups/myapp_$(date +%Y%m%d_%H%M%S).dump

# Dump all databases
pg_dumpall -h localhost -U postgres -f /backups/all_databases_$(date +%Y%m%d).sql

# Restore from a custom-format dump
pg_restore -h localhost -U postgres -d myapp -c /backups/myapp_20260325_020000.dump

For production workloads, pg_basebackup creates a physical backup of the entire cluster and can be combined with WAL archiving for point-in-time recovery (PITR):

1
2
3
4
5
6

# Physical base backup
pg_basebackup -h localhost -U replicator -D /backups/pgbase -Ft -z -P

# WAL archiving in postgresql.conf
archive_mode = on
archive_command = 'cp %p /backups/wal/%f'

With WAL archiving enabled, you can restore to any point in time — not just when you took the backup. This is how you recover from “someone ran DELETE without WHERE at 3:47pm and we need everything as it was at 3:46pm.”

MySQL / MariaDB

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Logical dump
mysqldump -u root -p --single-transaction --routines --triggers mydb \
  > /backups/mydb_$(date +%Y%m%d_%H%M%S).sql

# Compressed dump
mysqldump -u root -p --single-transaction mydb | gzip \
  > /backups/mydb_$(date +%Y%m%d).sql.gz

# Restore
mysql -u root -p mydb < /backups/mydb_20260325.sql

For hot backups without locking, Percona XtraBackup is the standard for InnoDB:

1
2

xtrabackup --backup --target-dir=/backups/mysql_base
xtrabackup --prepare --target-dir=/backups/mysql_base

Redis

Redis has two persistence modes. RDB (Redis Database) is point-in-time snapshots. AOF (Append Only File) logs every write operation and survives crashes better:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Trigger a background save
redis-cli BGSAVE

# Check last save time
redis-cli LASTSAVE

# In redis.conf — enable both RDB and AOF
save 900 1
save 300 10
appendonly yes
appendfsync everysec

For DR purposes, treat Redis as cache-tier unless your application absolutely depends on its data surviving a restart. If it’s storing sessions, queues, or rate limit state that must survive, treat it like a database.

MongoDB

1
2
3
4
5
6
7
8

# Dump all databases
mongodump --uri="mongodb://user:pass@localhost:27017" --out=/backups/mongo_$(date +%Y%m%d)

# Dump a single database
mongodump --uri="mongodb://user:pass@localhost:27017/mydb" --out=/backups/mydb_$(date +%Y%m%d)

# Restore
mongorestore --uri="mongodb://user:pass@localhost:27017" /backups/mongo_20260325/

Replica sets are also a form of DR for MongoDB — a secondary can be promoted in seconds if the primary fails.

Files and Object Storage with Restic

Restic is the best general-purpose backup tool for files. It’s encrypted at rest, deduplicated (so backing up the same files again costs almost nothing in storage), supports multiple backends, and has a consistent interface regardless of where you’re storing data.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# Initialize a repository (local)
restic init --repo /mnt/backup/myrepo

# Initialize on Backblaze B2
export B2_ACCOUNT_ID="your_account_id"
export B2_ACCOUNT_KEY="your_account_key"
restic init --repo b2:my-bucket:restic

# Run a backup
restic backup /home/jmoon /etc /opt/myapp

# List snapshots
restic snapshots

# Restore a specific snapshot
restic restore abc12345 --target /tmp/restore

# Restore a specific path from the latest snapshot
restic restore latest --target /tmp/restore --path /home/jmoon/documents

# Check repository integrity
restic check
restic check --read-data  # also verifies all data can be read (slower)

A production-ready Restic backup script with retention:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

#!/bin/bash
set -euo pipefail

# Configuration
REPO="b2:lunarops-backup:restic"
BACKUP_PATHS="/home /etc /opt/apps /var/lib/docker/volumes"
LOG_FILE="/var/log/restic-backup.log"
HEALTHCHECK_URL="https://hc-ping.com/your-uuid-here"

export B2_ACCOUNT_ID="your_account_id"
export B2_ACCOUNT_KEY="your_account_key"
export RESTIC_PASSWORD_FILE="/etc/restic/password"

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

log "Starting backup..."

# Run the backup
if restic backup \
    --repo "$REPO" \
    $BACKUP_PATHS \
    --exclude-file=/etc/restic/excludes \
    --tag "$(hostname)" \
    2>&1 | tee -a "$LOG_FILE"; then

    log "Backup completed successfully."

    # Apply retention policy
    log "Applying retention policy..."
    restic forget \
        --repo "$REPO" \
        --keep-daily 7 \
        --keep-weekly 4 \
        --keep-monthly 12 \
        --keep-yearly 3 \
        --prune \
        2>&1 | tee -a "$LOG_FILE"

    log "Pruning complete."

    # Ping healthcheck on success
    curl -fsS --retry 3 "$HEALTHCHECK_URL" > /dev/null
    log "Healthcheck pinged."
else
    log "ERROR: Backup failed!"
    # Ping healthcheck failure endpoint
    curl -fsS --retry 3 "${HEALTHCHECK_URL}/fail" > /dev/null
    exit 1
fi

Restic backends:

• local:/path/to/dir — local filesystem or NAS mount
• sftp:user@host:/path — SSH/SFTP (great for a second machine)
• b2:bucket:path — Backblaze B2
• s3:s3.amazonaws.com/bucket — AWS S3 or compatible (Wasabi, MinIO)
• rclone:remote:path — any rclone-supported backend

rclone for cloud sync:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Configure a remote (interactive)
rclone config

# Sync to B2
rclone sync /backups b2:my-backup-bucket/server1 \
    --progress \
    --transfers 8 \
    --checksum

# Copy with bandwidth throttle (useful for background sync)
rclone copy /backups b2:my-backup-bucket/server1 \
    --bwlimit 50M \
    --progress

VM and Container State

Proxmox vzdump:

1
2
3
4
5
6
7
8
9

# Backup a VM (ID 100) to a storage pool
vzdump 100 --storage local-zfs --compress zstd --mode snapshot

# Backup all VMs
vzdump --all --storage pbs --compress zstd --mode snapshot

# Restore a VM
qmrestore /var/lib/vz/dump/vzdump-qemu-100-2026_03_25-02_00_00.vma.zst 100 \
    --storage local-zfs

Proxmox Backup Server (PBS) adds deduplication, encryption, and incremental backups on top of vzdump — if you have a Proxmox cluster, PBS is worth setting up as your primary backup target.

Docker volume backups:

Named volumes don’t live in a path you can just rsync. The standard pattern for backing them up:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Back up a named volume to a tar archive
docker run --rm \
    -v myapp_data:/data:ro \
    -v /backups:/backup \
    busybox tar czf /backup/myapp_data_$(date +%Y%m%d).tar.gz -C /data .

# Restore a volume from a tar archive
docker volume create myapp_data_restored
docker run --rm \
    -v myapp_data_restored:/data \
    -v /backups:/backup \
    busybox tar xzf /backup/myapp_data_20260325.tar.gz -C /data

Configuration and Secrets: Cattle, Not Pets

If your configuration lives in git and your secrets are in a secrets manager, you don’t need to back up the VM. You can rebuild it from scratch in minutes. This is the “cattle, not pets” principle applied to disaster recovery.

SOPS for encrypted secrets in git:

1
2
3
4
5
6
7
8

# Encrypt a secrets file with age
sops --encrypt --age age1xxxxxxxxxx secrets.yaml > secrets.enc.yaml

# Add to git
git add secrets.enc.yaml

# Decrypt for use
sops --decrypt secrets.enc.yaml > secrets.yaml

HashiCorp Vault snapshots:

1
2
3
4
5

# Take a snapshot
vault operator raft snapshot save /backups/vault_$(date +%Y%m%d_%H%M%S).snap

# Restore (from a cold state)
vault operator raft snapshot restore /backups/vault_20260325_020000.snap

The 3-2-1-1-0 Backup Rule

The classic 3-2-1 rule says: keep 3 copies of your data, on 2 different storage media, with 1 copy offsite.

The extended 3-2-1-1-0 rule adds two more requirements: 1 copy offline or air-gapped (protecting against ransomware that encrypts everything reachable on the network), and 0 errors — meaning you verify your backups actually work.

In practice for a homelab:

1. Copy 1: Live data on your primary storage
2. Copy 2: Local NAS or backup drive (different media)
3. Copy 3: Cloud storage (Backblaze B2, Wasabi, S3) — offsite
4. Copy 4: Offline encrypted drive rotated periodically — air-gapped, ransomware-proof

Immutable backups against ransomware:

Backblaze B2 and AWS S3 both support Object Lock, which makes objects immutable for a specified period. Even if ransomware compromises your machine and has valid API credentials, it cannot delete or overwrite locked objects:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Enable Object Lock on a B2 bucket (done at bucket creation)
# Or use the B2 CLI:
b2 update-bucket --defaultRetentionMode compliance \
    --defaultRetentionPeriod 30days \
    my-backup-bucket

# For S3 Object Lock with Restic
aws s3api put-object-lock-configuration \
    --bucket my-restic-bucket \
    --object-lock-configuration '{"ObjectLockEnabled": "Enabled", "Rule": {"DefaultRetention": {"Mode": "COMPLIANCE", "Days": 30}}}'

With a 30-day retention lock, an attacker who compromises your credentials can still write new objects but cannot delete objects that are less than 30 days old. Your backups survive.

Writing Runbooks

A runbook is a documented, step-by-step procedure for responding to a specific failure scenario. It is not a general guide. It is not a “how Postgres works” explainer. It is: here is the exact situation, here are the exact steps to execute, here is how you verify you’re done.

Anatomy of a Good Runbook

Every runbook should have the same structure:

# Runbook: [Descriptive Title of the Failure Scenario]

## Metadata
- Severity: [Critical / High / Medium / Low]
- Service: [What service is affected]
- Owner: [Team or person responsible]
- Last reviewed: [Date]
- Last tested: [Date and by whom]

## Impact
[What the user-visible impact is. Be specific: "Users cannot log in" not "auth is down"]

## Detection
[How you know this situation is happening. Alert name, dashboard link, error message]

## Prerequisites
- [ ] Access to [specific system] via [method]
- [ ] [Tool] installed on your workstation
- [ ] [Credentials/keys] from [location]

## Recovery Procedure

### Step 1: [First action]
[Explanation of why this step comes first]
```bash
command to run

Expected output: [what you should see]

Step 2: [Next action]

…

Verification

[Specific checks that confirm recovery is complete]

1
2

# Test command
curl -f https://myapp.example.com/health

Expected: HTTP 200 with {"status": "ok"}

Escalation

If recovery fails after [X] minutes, contact [person/team] via [method].

Post-Incident

• Update this runbook with any gaps found
• File an incident report
• Schedule a postmortem if severity >= High

### Runbook: Database Corruption

Runbook: PostgreSQL Database Corruption / Data Loss

Metadata

• Severity: Critical
• Service: PostgreSQL (myapp database)
• Owner: Platform team
• Last reviewed: 2026-03-25
• Last tested: 2026-02-15 by jmoon

Impact

Application is throwing database errors. Data may be missing or inconsistent. Users may see errors, missing content, or failed transactions.

Detection

• Alert: “Postgres query errors > 5% for 5 minutes”
• OR: Application logs showing: ERROR: invalid page in block X of relation
• OR: pg_catalog.pg_stat_database.checksum_failures > 0

Prerequisites

• SSH access to db-primary.internal
• psql access as postgres user (or pgpass configured)
• Access to /backups/postgres/ on backup NAS
• Downtime window approved (coordinate with team in #incidents)

Recovery Procedure

Step 1: Stop the application (prevent further writes to corrupted data)

1
2
3
4

# On app servers
sudo systemctl stop myapp
# Or for Docker
docker compose stop myapp

Verify no connections to the database (except your own):

1

psql -U postgres -c "SELECT count(*) FROM pg_stat_activity WHERE datname='myapp';"

Step 2: Identify the extent of corruption

1
2
3
4
5

# Check for checksum failures (if data checksums enabled)
psql -U postgres -c "SELECT datname, checksum_failures FROM pg_stat_database WHERE datname='myapp';"

# Run pg_dump to see if the database can be read
pg_dump -U postgres -d myapp --schema-only > /dev/null 2>&1 && echo "Schema OK" || echo "Schema errors"

Step 3: Identify the last good backup

1
2
3
4
5

# List available backups
ls -lth /backups/postgres/*.dump | head -20

# Check backup integrity
pg_restore --list /backups/postgres/myapp_YYYYMMDD_HHMMSS.dump > /dev/null && echo "Backup OK"

If using Restic: restic snapshots --tag postgres --repo /mnt/backup/restic

Step 4: Create a snapshot of current (corrupted) state

1
2
3

# Never throw away the corrupted data until recovery is confirmed
pg_dump -U postgres -d myapp -F c \
  -f /backups/postgres/myapp_corrupted_$(date +%Y%m%d_%H%M%S).dump || true

Step 5: Drop and recreate the database

1
2

psql -U postgres -c "DROP DATABASE myapp;"
psql -U postgres -c "CREATE DATABASE myapp OWNER myapp_user;"

Step 6: Restore from backup

1
2

pg_restore -U postgres -d myapp -c \
  /backups/postgres/myapp_YYYYMMDD_HHMMSS.dump

Watch for errors in the output. Warnings about ownership are usually safe to ignore.

Step 7: Verify data integrity

1
2
3
4

# Run application-specific checks
psql -U postgres -d myapp -c "SELECT count(*) FROM users;"
psql -U postgres -d myapp -c "SELECT max(created_at) FROM orders;"
# Compare counts against known-good values or yesterday's metrics

Step 8: Restart the application

1
2
3

sudo systemctl start myapp
# Tail logs for 2-3 minutes watching for errors
sudo journalctl -u myapp -f

Verification

• Application health endpoint returns 200: curl -f https://myapp.example.com/health
• Login works end-to-end
• Data from before the corruption point is present
• Error rate in monitoring has dropped to baseline

Post-Incident

• Identify the root cause of corruption (hardware? bug? power loss?)
• Calculate actual data loss window (time of last good backup to incident)
• Notify affected users if data was lost
• Update runbook with anything that was unclear

### Runbook: VM Host Failure

Runbook: Proxmox Host Node Failure

Metadata

• Severity: Critical
• Service: All VMs on affected Proxmox node
• Last tested: 2026-01-20

Impact

All virtual machines on [failed-node] are unavailable. Depending on which VMs are affected, multiple downstream services may be down.

Detection

• Alert: “Host failed-node unreachable for > 5 minutes”
• OR: VMs not responding to ping/HTTP checks
• OR: Proxmox cluster shows node as offline

Prerequisites

• Access to Proxmox web UI or pve node CLI (alternate node)
• Access to Proxmox Backup Server or NAS backup storage
• List of VMs on the failed node and their IDs (from CMDB or pvesh get /nodes/failed-node/qemu)

Recovery Procedure

Step 1: Confirm the failure and identify affected VMs

1
2
3
4

# From another Proxmox node
pvesh get /nodes/failed-node/qemu 2>/dev/null || echo "Node unreachable"
# Check cluster status
pvecm status

Step 2: For each critical VM — restore from latest backup

1
2
3
4
5
6
7
8

# List available backups on PBS
proxmox-backup-client list --repository pbs@pam@pbs.internal:backup-store

# Restore VM 101 to alternate-node, with new ID if needed
qmrestore PBS:backup/vzdump-qemu-101-2026_03_24-02_00_00.vma.zst 101 \
    --storage local-zfs \
    --node alternate-node \
    --start 1

Step 3: Restore in tier order

Restore Tier 0 services first (DNS, auth, monitoring), then Tier 1, etc. Do not move to the next tier until the current tier’s services are verified.

Step 4: Update DNS / load balancer records if IPs changed

1
2
3

# If using Pi-hole for local DNS
# Update /etc/pihole/custom.list with new IP
# Reload: pihole restartdns

Step 5: Notify users

Post in #status channel: “We are experiencing an infrastructure incident. [Affected services] are being restored. ETA: [X] minutes.”

Verification

For each restored VM:

• SSH access works
• Service is running (systemctl is-active <service>)
• Application responds to health checks
• Metrics appear in Grafana

### Runbook: Ransomware Response

Runbook: Ransomware Incident Response

Metadata

• Severity: Critical — treat as a security incident, not just an outage
• This runbook assumes ransomware has been confirmed or is strongly suspected

STOP. READ THIS FIRST.

Do NOT attempt to recover data or remove the ransomware while affected systems are still on the network. Isolation comes first, always.

Step 1: Isolate immediately

1
2
3
4
5
6
7
8
9

# Cut network access to affected hosts. Physical disconnection is preferred.
# If remote, block at firewall level immediately:
# (On pfSense/OPNsense: Firewall > Rules > add block rule for affected IPs)

# On the affected host (if you have access):
ip link set eth0 down
# Or block all traffic:
iptables -I INPUT -j DROP
iptables -I OUTPUT -j DROP

Step 2: Identify the blast radius

• Which systems are encrypted or behaving abnormally?
• What shares, volumes, and backup destinations were mounted on affected systems?
• Timeline: when did the behavior start? (check logs, file modification timestamps)

Step 3: DO NOT REBOOT affected systems yet

Rebooting may destroy forensic evidence. Capture volatile data first if you have the skills:

1
2
3
4

# Capture running processes, network connections, loaded modules
ps auxf > /tmp/processes.txt
ss -tulpn > /tmp/connections.txt
lsmod > /tmp/modules.txt

Step 4: Identify last clean offline backup

Your offline/air-gapped backup is your lifeline here. Network-connected backups may also have been encrypted if they were reachable.

1
2
3
4
5

# Restic — list snapshots, find the one before the incident
restic snapshots --repo /path/to/offline-repo --tag myserver

# Check file modification times to identify infection window
find /data -newer /tmp/reference_file -type f | head -50

Step 5: Restore from clean backup

• Wipe and rebuild affected systems from scratch (do not trust anything that was running)
• Restore data only from backups predating the infection timeline
• Verify restored data before bringing systems online

Step 6: Rotate ALL credentials

Everything that was accessible from compromised systems must be treated as compromised:

• All SSH keys
• All API tokens and service account passwords
• All secrets in Vault or similar (rotate before restoring from Vault snapshot)
• Cloud provider credentials
• Database passwords
• Any credentials stored in config files

Step 7: Harden before reconnecting

• Patch the vulnerability that was exploited (if known)
• Review firewall rules — what should not have been reachable?
• Enable network monitoring before bringing systems back online

Post-Incident

• Full incident timeline documentation
• Root cause analysis: how did attacker gain access?
• Regulatory/legal notification if personal data was involved
• Blameless postmortem with the team

### Where to Store Runbooks

Your runbooks must be accessible when your systems are down. That means:

- **Primary**: In git, in the same repository as your infrastructure code
- **Secondary**: A wiki platform (Notion, Confluence, Obsidian + sync) accessible from outside your infrastructure
- **Tertiary**: Printed or exported to PDF on a USB drive in your server rack

The **2am test**: Hand your runbook to a competent engineer who didn't write it. Can they execute it successfully at 2am, half-asleep, without asking you a single question? If not, it needs more detail.

---

## Chaos Engineering: Finding Weaknesses Before Production Does

Chaos engineering is the practice of intentionally introducing failures into your systems to verify they behave as expected and to discover weaknesses before an unplanned outage does it for you.

Netflix famously ran Chaos Monkey in production, randomly terminating instances to prove their services could survive it. For most organizations, controlled fault injection in a staging or homelab environment is the more appropriate starting point.

The core principle: **the only way to know your system survives a failure is to make it fail on your terms.**

### Tools

**Chaos Mesh** (Kubernetes-native)

Install via Helm:

```bash
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
    --namespace=chaos-mesh \
    --create-namespace \
    --version 2.7.0

Kill a random pod every 5 minutes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-kill-example
  namespace: chaos-mesh
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - default
    labelSelectors:
      "app": "myapp"
  scheduler:
    cron: "@every 5m"

Inject network latency and packet loss:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: network-delay-example
  namespace: chaos-mesh
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - default
    labelSelectors:
      "app": "myapp"
  delay:
    latency: "100ms"
    correlation: "25"
    jitter: "20ms"
  loss:
    loss: "20"
    correlation: "25"
  duration: "10m"

Fill a disk to trigger out-of-space conditions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

apiVersion: chaos-mesh.org/v1alpha1
kind: IOChaos
metadata:
  name: disk-fill-example
  namespace: chaos-mesh
spec:
  action: disk-fill
  mode: one
  selector:
    namespaces:
      - default
    labelSelectors:
      "app": "myapp"
  size: "90%"
  path: "/data"
  duration: "5m"

Pumba (Docker chaos)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Kill a container randomly every 30 seconds
pumba --random kill --interval 30s "re2:myapp_.*"

# Add 200ms delay with 20ms jitter to a container's network
pumba netem --duration 5m delay \
    --time 200 \
    --jitter 20 \
    myapp_web_1

# Pause a container (simulates a frozen/unresponsive process)
pumba pause --duration 30s myapp_db_1

Linux traffic control (tc netem) — no special tools required:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Add 150ms delay + 30ms jitter to eth0 (simulates a bad network link)
tc qdisc add dev eth0 root netem delay 150ms 30ms

# Add packet loss
tc qdisc add dev eth0 root netem loss 10%

# Add packet corruption
tc qdisc add dev eth0 root netem corrupt 1%

# Remove the effect
tc qdisc del dev eth0 root

# List current rules
tc qdisc show dev eth0

stress-ng for resource pressure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# CPU stress: 4 workers for 60 seconds
stress-ng --cpu 4 --timeout 60s

# Memory pressure: allocate 2GB
stress-ng --vm 2 --vm-bytes 2G --timeout 60s

# Disk I/O pressure
stress-ng --io 4 --hdd 2 --timeout 60s

# Combined: simulate a resource-constrained system
stress-ng --cpu 2 --vm 1 --vm-bytes 1G --io 2 --timeout 120s

Running a Game Day

A game day is a scheduled, time-boxed chaos exercise. It’s not random — it’s structured.

How to run one:

1. Define scope and hypothesis — What are we testing? What do we expect to happen?

   • Example: “Kill the primary database. We expect the application to detect failure within 30 seconds and fail over to the replica within 2 minutes. User-visible errors should last no longer than 3 minutes.”

2. Define success and abort criteria — What does a pass look like? When do we stop?

   • Success: Failover completes within 2 minutes, no data loss
   • Abort: If failover hasn’t completed within 10 minutes, restore manually

3. Notify stakeholders — Even in staging, let people know.

4. Execute the experiment — One person runs the chaos. Others observe metrics, logs, and user experience.

5. Document observations in real time — Don’t rely on memory.

6. Stop and restore — Return to a known-good state.

7. Debrief — What happened vs. what you expected? What do you need to fix?

Example game day scenarios:

• “Kill the primary database — how long to fail over? Do alerts fire? Does the application recover gracefully or do we need a manual restart?”
• “Fill the disk on the web server — do alerts fire with enough lead time? Does the application degrade gracefully or does it start throwing 500s silently?”
• “Disconnect the NAS — what services break immediately? What degrades gradually? What keeps working?”
• “Revoke the API key your app uses — does it fail with a clear error or silently do nothing?”
• “Simulate a slow database — inject 2 second latency on db queries — do timeouts kick in or does the app hang?”

Starting without dedicated chaos tools:

You don’t need Chaos Mesh to start. The most valuable experiments you can run today require nothing but your existing access:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Kill a container
docker stop myapp_web_1

# Kill a process hard
kill -9 $(pgrep postgres)

# Fill a disk
fallocate -l 10G /var/tmp/diskfiller.bin

# Unplug (simulate) a network interface
ip link set eth1 down

# Corrupt a config file
echo "bad config" >> /etc/nginx/nginx.conf && nginx -s reload

Run the experiment, observe what happens, put it back.

DR Testing: Making It Real

Testing your DR plan is not optional. It is the plan. A DR plan that has never been tested is a document, not a plan.

The Restore Test Schedule

Monthly file restore test:

Pick a random file that was backed up. Restore it to a temporary directory. Verify it matches the original.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

#!/bin/bash
# Monthly spot-check: restore a random file and verify integrity

REPO="b2:lunarops-backup:restic"
RESTORE_DIR="/tmp/restore_test_$(date +%Y%m%d)"

# Pick a random snapshot
SNAPSHOT=$(restic snapshots --repo "$REPO" --json | \
    jq -r '.[].id' | shuf | head -1)

echo "Testing snapshot: $SNAPSHOT"

# Get a list of files and pick one randomly
TEST_FILE=$(restic ls --repo "$REPO" "$SNAPSHOT" --json | \
    jq -r 'select(.type=="file") | .path' | \
    grep -E '\.(conf|sql|yaml|json|md)$' | \
    shuf | head -1)

echo "Restoring: $TEST_FILE"

restic restore "$SNAPSHOT" \
    --repo "$REPO" \
    --target "$RESTORE_DIR" \
    --path "$TEST_FILE"

# Verify the restored file is readable and non-empty
if [ -s "${RESTORE_DIR}${TEST_FILE}" ]; then
    echo "PASS: File restored successfully ($(wc -c < "${RESTORE_DIR}${TEST_FILE}") bytes)"
else
    echo "FAIL: Restored file is empty or missing"
    exit 1
fi

# Cleanup
rm -rf "$RESTORE_DIR"

Quarterly application restore test:

1. Provision a clean test VM or Docker environment
2. Restore the latest application backup to it (database, volumes, config)
3. Start the application stack
4. Run smoke tests against it (automated or manual)
5. Measure how long the entire process took — this is your actual RTO
6. Document gaps and clean up

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# Example: restore a Docker Compose stack from Restic
RESTORE_DIR="/tmp/dr-test-$(date +%Y%m%d)"
mkdir -p "$RESTORE_DIR"

# Restore app volumes and config
restic restore latest \
    --repo "$REPO" \
    --target "$RESTORE_DIR" \
    --path /opt/myapp

# Restore database dump
restic restore latest \
    --repo "$REPO" \
    --target "$RESTORE_DIR" \
    --path /backups/postgres/myapp_latest.dump

# Bring up stack against test database
cd "$RESTORE_DIR/opt/myapp"
export DATABASE_HOST=localhost
docker compose up -d

# Run smoke tests
sleep 30
curl -f http://localhost:8080/health && echo "PASS" || echo "FAIL"

Automated Backup Integrity Verification

Add this as a weekly cron job:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

#!/bin/bash
# Weekly: verify backup repository integrity
set -euo pipefail

REPO="b2:lunarops-backup:restic"
LOG="/var/log/restic-check.log"
HEALTHCHECK_URL="https://hc-ping.com/your-weekly-check-uuid"

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG"; }

log "Starting weekly backup integrity check..."

if restic check --repo "$REPO" --read-data-subset=10% 2>&1 | tee -a "$LOG"; then
    log "Integrity check passed."
    curl -fsS "$HEALTHCHECK_URL" > /dev/null
else
    log "ERROR: Integrity check failed!"
    curl -fsS "${HEALTHCHECK_URL}/fail" > /dev/null
    # Send alert (mail, Slack webhook, etc.)
    mail -s "BACKUP INTEGRITY CHECK FAILED" admin@example.com < "$LOG"
    exit 1
fi

The Gap Between Documented and Actual RTO

Every organization discovers the same thing when they run their first real DR exercise: it takes longer than the documented RTO. Always.

Common reasons:

• The person who knows the restore procedure is on vacation
• The runbook references a file path that changed six months ago
• The test environment doesn’t have the same version of the restore tool
• The backup credential expired and nobody noticed
• The restore process works fine but the application won’t start because of a config file that wasn’t backed up
• DNS still points at the old IP after the restore

Measure your actual RTO during every test. Track it over time. If your documented RTO is 2 hours and your actual RTO is 8 hours, that’s your real RTO until you fix the gaps.

Monitoring for DR Readiness

The best time to know your backup job stopped running is not when you need it. These alerts tell you disaster is coming, not that it has arrived.

Proactive Alerts

Disk filling up — predict before it’s full:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Prometheus alerting rule
- alert: DiskFillPredicted24h
  expr: |
    predict_linear(node_filesystem_free_bytes{mountpoint="/"}[6h], 24 * 3600) < 0
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "Disk on {{ $labels.instance }} predicted to fill in 24h"
    description: "At the current write rate, {{ $labels.mountpoint }} will be full within 24 hours."

Backup job hasn’t run:

1
2
3
4
5
6
7

- alert: BackupJobMissed
  expr: time() - backup_last_success_timestamp > 90000  # 25 hours
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Backup job has not completed successfully in 25 hours"

RAID degraded:

1
2
3
4
5
6
7

- alert: RAIDDegraded
  expr: node_md_disks{state="failed"} > 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "RAID array {{ $labels.device }} has failed disks"

Replication lag too high:

1
2
3
4
5
6
7

- alert: PostgresReplicationLagHigh
  expr: pg_replication_lag > 300  # 5 minutes
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "PostgreSQL replication lag is {{ $value }}s"

Dead Man’s Switch Monitoring

A dead man’s switch inverts the normal alerting model. Instead of alerting when something bad happens, it alerts when something stops happening. Perfect for backup jobs.

Healthchecks.io (free tier available) or self-hosted with Uptime Kuma:

1
2
3
4
5
6
7

# At the end of your backup script, ping the healthcheck URL
# If no ping is received within the check period, an alert fires

curl -fsS --retry 3 https://hc-ping.com/your-unique-uuid > /dev/null

# On failure, ping the /fail endpoint
curl -fsS --retry 3 https://hc-ping.com/your-unique-uuid/fail > /dev/null

Configure the check with a period of 24 hours and a grace period of 1 hour. If your daily backup job doesn’t ping within 25 hours, you get notified before you need the backup, not after.

DR for Common Homelab Stacks

Proxmox + Proxmox Backup Server

PBS is the right backup target for Proxmox. It deduplicates across VMs and snapshots, supports encryption, and integrates directly with the Proxmox scheduler.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# Add PBS as a storage target (in Proxmox web UI, or via CLI)
pvesm add pbs pbs-storage \
    --server pbs.internal \
    --datastore backup-store \
    --username backup@pam \
    --fingerprint AA:BB:CC:...

# Schedule backups for all VMs, daily at 2am
# In Proxmox web UI: Datacenter > Backup > Add
# Or via API:
pvesh create /cluster/backup \
    --storage pbs-storage \
    --schedule "0 2 * * *" \
    --mode snapshot \
    --compress zstd \
    --all 1

# Sync PBS repository to cloud with rclone (run on PBS host)
rclone sync /var/lib/proxmox-backup/datastore/backup-store \
    b2:lunarops-pbs-offsite/backup-store \
    --progress \
    --transfers 4

Docker Compose Stack

The target state: your entire stack can be rebuilt from git + restored volumes in one session.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# DR restore script for a Docker Compose stack
#!/bin/bash
set -euo pipefail

APP_DIR="/opt/myapp"
RESTIC_REPO="b2:lunarops-backup:restic"

echo "Step 1: Pull latest config from git"
git clone https://git.internal/infra/myapp-stack.git "$APP_DIR"

echo "Step 2: Restore data volumes from Restic"
for volume in myapp_db_data myapp_app_data myapp_redis_data; do
    echo "Restoring volume: $volume"
    docker volume create "$volume"
    restic restore latest \
        --repo "$RESTIC_REPO" \
        --target /tmp/vol-restore \
        --path "/var/lib/docker/volumes/$volume/_data"
    docker run --rm \
        -v "$volume":/target \
        -v /tmp/vol-restore:/source \
        busybox cp -a /source/. /target/
done

echo "Step 3: Start the stack"
cd "$APP_DIR"
docker compose up -d

echo "Step 4: Run health checks"
sleep 30
curl -f http://localhost/health && echo "Stack is healthy" || echo "HEALTH CHECK FAILED"

K3s Cluster

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# Etcd snapshot (K3s embedded etcd)
k3s etcd-snapshot save --name pre-upgrade-$(date +%Y%m%d)

# List snapshots
k3s etcd-snapshot list

# Restore (requires stopping k3s first)
systemctl stop k3s
k3s etcd-snapshot restore --name pre-upgrade-20260325

# Velero for PVC backup
velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.9.0 \
    --bucket lunarops-velero \
    --secret-file ./credentials-velero \
    --backup-location-config region=us-west-002,s3ForcePathStyle=true,s3Url=https://s3.us-west-002.backblazeb2.com

# Create a scheduled backup
velero schedule create daily-backup \
    --schedule="0 3 * * *" \
    --ttl 720h

# Restore from a backup
velero restore create --from-backup daily-backup-20260325030000

Keep your Helm values files and all Kubernetes manifests in git. If you can kubectl apply -f your entire cluster from scratch, your RTO for cluster rebuild is as short as your provisioning scripts allow.

Home Assistant

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# Home Assistant creates backups automatically — also do this via CLI
ha backup new --name "pre-update-$(date +%Y%m%d)"

# Copy latest backup off the HA host
rsync -av homeassistant.local:/backup/latest.tar /backups/homeassistant/

# Better: mount a NAS share in HA and point backups there
# In configuration.yaml:
# homeassistant:
#   backup:
#     media_dir: /media/nas_backup

# Sync config directory to git (sanitized — remove secrets)
# Store the full config (with secrets) encrypted via git-crypt or SOPS

The most important HA backup: the configuration.yaml, automations, scripts, and custom components. The history database (home-assistant_v2.db) is large and less critical — define your RPO for it separately.

Putting It Together: Your DR Readiness Checklist

Use this as a self-assessment. Each “no” is a gap in your DR posture.

Documentation

• Service inventory exists and is current
• Dependency map exists for Tier 0 and Tier 1 services
• RTO/RPO targets are defined for each service tier
• Runbooks exist for your top 5 failure scenarios
• Runbooks are stored in at least 2 locations, one accessible without your infrastructure

Backups

• All Tier 0 and Tier 1 data is backed up on a schedule meeting RPO targets
• Backups are encrypted
• At least one offsite backup exists
• At least one offline or immutable backup exists (ransomware protection)
• Backup jobs are monitored (dead man’s switch, alerting on failure)

Testing

• You have restored from backup in the last 30 days (any file)
• You have run a full application restore test in the last 90 days
• You have measured actual RTO and compared it to your target
• Your runbooks have been executed by someone who didn’t write them

Monitoring

• Disk fill prediction alerts are configured
• Backup job missed alerts are configured
• RAID/storage health alerts are configured
• You have a status page or incident communication channel

Chaos

• You have intentionally failed at least one service in the last quarter and observed recovery
• You know what breaks when your most critical dependency goes down

Disaster recovery is not a one-time project. It’s a practice — something you do regularly, test relentlessly, and improve incrementally. The backup job you set up and forgot about three years ago is not your safety net. The runbook you wrote but never tested is not your plan.

Your DR strategy is exactly as strong as the last time you actually used it. So use it. Schedule a restore test this week. Run a game day next month. Find the gaps before they find you at 2am.