Backup Strategy in Practice: Restic, Backblaze B2, rclone, and Actually Testing Restores

Most people set up backups after their first data loss. That is the wrong order.

The hard part is not the tooling — the tooling is excellent and mostly free. The hard part is convincing yourself, before anything bad happens, that the hour spent configuring and testing a backup system is worth more than the hour spent building the next feature, adding the next service, or doing anything else that feels more immediately productive. It is. By a large margin.

This guide covers the full picture: the philosophy behind a real backup strategy, the 3-2-1 rule and its modern extensions, Restic as the backbone for encrypted deduplicated backups, Backblaze B2 as the best-value cloud storage target, rclone for multi-cloud flexibility, complete production automation scripts, healthcheck monitoring, and — most importantly — how to verify your backups actually work before you need them.

Part 1: The Backup Mindset

Before any tooling, some hard truths.

It’s Not a Backup Until You’ve Tested the Restore

This phrase gets repeated so often it risks becoming a cliche, but it keeps being true. Backup software has bugs. Filesystems have corruption. Repositories get partial writes. The only way to know your backup is valid is to restore from it and verify the result.

There are three distinct failure modes for backups:

1. No backup exists. Obvious, but more common than it should be. “I meant to set that up.”
2. A backup exists but is unrestorable. The worst failure mode because it creates false confidence. Your backup job ran for six months, but the repository password is wrong, or the files are zero bytes, or a software bug silently corrupted the data.
3. A backup exists and is valid, but recovery takes too long. You have a working restore process, but it takes 18 hours and your business can only survive 4 hours of downtime. The backup technically works but it fails your recovery objective.

All three kill you. Testing addresses the second and surfaces the third.

Recovery Objectives: RTO and RPO

The third failure mode — a valid backup that recovers too slowly — is really a failure to plan against two numbers. Every backup strategy is implicitly a bet on them, so make the bet explicit:

• RPO (Recovery Point Objective): how much data you can afford to lose, measured in time. If you back up once a day, your RPO is up to 24 hours — a disaster at 1:59am, one minute before the 2am run, loses almost a full day of changes. If you cannot tolerate that, you need more frequent backups or continuous replication.
• RTO (Recovery Time Objective): how long you can afford to be down. This is the wall-clock time from “disaster” to “service restored,” including provisioning hardware, downloading the backup, importing databases, and bringing applications up. A backup that takes 18 hours to restore fails a 4-hour RTO no matter how valid the data is.

RPO drives backup frequency; RTO drives backup architecture. The cheaper your recovery needs to be, the longer it takes — restoring 2TB from cold cloud storage is slower than failing over to a warm standby. Decide the targets first, then build backwards to a schedule and topology that meet them:

Part 8 covers how to measure your real RTO during a restore drill — the number you assume is almost always optimistic.

Backup vs. Sync: A Critical Distinction

Sync tools — rsync, cloud sync clients, RAID — are not backups. They are mirrors. When you delete a file, the mirror deletes it too. When ransomware encrypts your data, the sync propagates the encrypted versions to the mirror within minutes.

A backup is a separate, point-in-time copy that retains historical versions. If you accidentally delete /var/lib/postgres today, you need to restore from a snapshot made before the deletion. A sync would have already destroyed that snapshot.

This matters in practice: if your “backup” is a cron job that rsyncs to a NAS, and that NAS is always mounted as a network share, ransomware will encrypt both in the same sweep. You need immutable, versioned, offline copies.

Why Cloud Storage Alone Isn’t a Backup

Storing data in Dropbox, Google Drive, or S3 is not a backup strategy. Cloud providers expose you to:

• Accidental deletion: You delete a file; the change syncs everywhere
• Account termination: ToS violations, billing failures, or provider-side errors can lock or delete your data with minimal notice
• Ransomware propagation: Sync clients faithfully upload your encrypted files
• No historical versions (unless you pay for version history, which is not default everywhere)

Cloud storage is a component of a backup strategy. It is not a substitute for one.

Part 2: The 3-2-1 Rule and Its Modern Extensions

The 3-2-1 rule is decades old but still the clearest mental model for backup architecture.

Original 3-2-1:

• 3 total copies of your data
• 2 different storage media types
• 1 copy offsite

Applied practically for a home server or VPS:

The offsite copy is what protects you from physical disasters: fire, flood, theft, or a power event that takes out all your local hardware simultaneously.

The 3-2-1-1-0 Extension

The modern extension adds two more requirements:

• +1 offline or air-gapped copy (not reachable over the network during normal operation)
• +0 errors verified (the backup has been validated, not just written)

The offline copy protects against ransomware that can reach network-attached storage. If your backup destination is always mounted, it can be encrypted along with the rest of your system. An offline copy — a drive that is physically disconnected between backup runs, or a cloud bucket with Object Lock enabled — cannot be encrypted by malware that doesn’t have those credentials.

Backblaze B2 Object Lock (compliance mode) is the practical implementation of the air-gap for cloud backups. Once data is written with a lock period, it cannot be deleted or overwritten — not by you, not by an attacker who compromises your backup credentials, not by Backblaze — until the lock expires.

Offline Drive Rotation

For a truly air-gapped local copy, a simple rotation schedule works:

• Keep 2-3 external drives labeled A, B, C
• Each week or month: connect the next drive in rotation, run a backup, disconnect it, store it offsite (or in a fireproof safe)
• The drive not currently connected is your offline copy

This sounds like overhead but takes about 15 minutes per cycle and protects against scenarios that no cloud solution addresses.

Immutable Backups with Object Lock

When configuring a Backblaze B2 bucket for Restic, enable Object Lock in compliance mode:

Lock mode: Compliance (not Governance — governance can be overridden by admin)
Default retention: 30 days (adjust based on your retention policy)

With compliance mode active, even a compromised application key with full write permissions to the bucket cannot delete objects within the lock period. This is the cloud equivalent of a write-once tape.

Full, Incremental, and Differential Backups

Classic backup theory defines three strategies, and it is worth understanding them because every backup tool is an answer to the trade-off between them:

• Full backup: a complete copy of all data every run. Simplest to restore (one copy has everything) but the slowest to create and the most storage-hungry.
• Incremental backup: only the data that changed since the previous backup of any kind. Fast and small to write, but a restore must replay the last full backup plus every incremental in the chain — and if one link is missing or corrupt, the chain breaks.
• Differential backup: only the data that changed since the last full backup. A restore needs just the full plus the single latest differential, so recovery is simpler than incremental, but each differential grows larger until the next full.

Restic — the backbone of this guide — sidesteps the choice. Its content-addressed deduplication means every snapshot is logically a full backup (you can restore any one directly, with no chain to replay) while being physically incremental (only changed chunks are uploaded and stored). You get the restore simplicity of full backups with the speed and storage efficiency of incrementals, and no fragile chain to corrupt. That is why the rest of this post stops talking about the taxonomy and starts talking about snapshots.

Part 3: Restic — Encrypted, Deduplicated Backups

Restic is the best general-purpose backup tool available today. It is open source, written in Go (single binary, no dependencies), and its design choices are exactly right.

What Makes Restic Excellent

Encryption by default. Every backup is encrypted with AES-256-CTR before leaving your system. There is no option to create an unencrypted repository. Your cloud provider, anyone with access to the bucket, and Backblaze employees cannot read your data.

Content-addressed deduplication. Restic splits files into variable-size chunks and addresses each chunk by its SHA-256 hash. If a 10GB virtual machine image changes by 100MB, only the changed chunks are uploaded — not the whole 10GB. Across snapshots, identical chunks are stored exactly once. This makes storage efficient and keeps backup times fast after the initial run.

Any backend. Local filesystem, SFTP, S3 and S3-compatible (B2, Wasabi, MinIO), Azure Blob Storage, Google Cloud Storage, or any provider accessible via rclone. One tool for all destinations.

Snapshots, not file copies. Each restic backup creates a snapshot — a point-in-time consistent view of the backed-up data. You can restore any snapshot, browse its contents, mount it as a filesystem, and diff between snapshots.

Installation

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# Debian/Ubuntu
sudo apt install restic

# Fedora/RHEL/Rocky
sudo dnf install restic

# macOS
brew install restic

# Download binary directly (always latest)
curl -L https://github.com/restic/restic/releases/latest/download/restic_linux_amd64.bz2 | bunzip2 > /usr/local/bin/restic
chmod +x /usr/local/bin/restic

# Self-update
restic self-update

Initial Repository Setup

Store credentials in an environment file rather than passing them on the command line:

1
2
3
4
5

# /etc/restic/env — mode 600, owned by root
export RESTIC_REPOSITORY="s3:s3.us-west-004.backblazeb2.com/your-bucket-name"
export RESTIC_PASSWORD="your-long-random-repository-password"
export AWS_ACCESS_KEY_ID="your-b2-key-id"
export AWS_SECRET_ACCESS_KEY="your-b2-application-key"

Generate a strong password: openssl rand -base64 32

Store this password somewhere safe and separate from the backup itself. A password manager, a printed copy in a fireproof safe, or a secrets manager. If you lose the repository password, the backup is permanently unreadable.

Initialize the repository:

1
2
3

source /etc/restic/env
restic init
# Repository created. Note the repository ID — store it with your password.

Core Commands

Back up a directory:

1
2

restic backup /data/important
restic backup /home /etc /var/lib/docker/volumes

Back up with excludes:

1
2
3
4
5
6
7

restic backup \
  --exclude='*.tmp' \
  --exclude='*.log' \
  --exclude='/home/*/.cache' \
  --exclude='/home/*/node_modules' \
  --exclude-file=/etc/restic/excludes \
  /home

Back up stdin (database dumps, generated archives):

1
2
3
4
5
6
7

pg_dump mydb | restic backup --stdin --stdin-filename postgres-mydb.sql

# MySQL / MariaDB
mysqldump --single-transaction mydb | restic backup --stdin --stdin-filename mysql-mydb.sql

# MongoDB (stream the archive format straight to stdin)
mongodump --db mydb --archive | restic backup --stdin --stdin-filename mongo-mydb.archive

List snapshots:

1
2
3
4

restic snapshots
restic snapshots --json | jq '.[].time'
restic snapshots --tag databases
restic snapshots --host myserver

Browse a snapshot:

1
2
3

restic ls latest
restic ls latest /home/user
restic ls 3a7c9d1f   # by snapshot ID (short IDs work)

Mount the repository as a filesystem (requires FUSE):

1
2
3
4
5

mkdir -p /mnt/restic
restic mount /mnt/restic &
ls /mnt/restic/snapshots/latest/
# Browse, diff, copy individual files
fusermount -u /mnt/restic

Full restore:

1
2

restic restore latest --target /tmp/restore
restic restore 3a7c9d1f --target /tmp/restore   # specific snapshot

Partial restore (specific paths):

1
2
3
4

restic restore latest \
  --include /home/user/.config \
  --include /home/user/.ssh \
  --target /tmp/restore

Diff between snapshots:

1
2

restic diff HEAD~1 latest
restic diff 3a7c9d1f 5b8e2a0c

Retention Policy: forget and prune

Restic accumulates snapshots over time. The forget and prune commands manage retention.

• forget removes snapshot metadata according to your policy but does not delete data
• prune removes unreferenced data chunks from the repository

Run them together:

1
2
3
4
5
6
7

restic forget \
  --keep-last 3 \
  --keep-daily 7 \
  --keep-weekly 4 \
  --keep-monthly 12 \
  --keep-yearly 3 \
  --prune

This keeps: the last 3 snapshots regardless of age, one snapshot per day for the past 7 days, one per week for 4 weeks, one per month for a year, and one per year for 3 years. Everything else is removed.

Add --dry-run to see what would be removed before committing.

If you have multiple hosts or tags in the same repository, scope forget to them:

1
2

restic forget --host myserver --tag files --keep-daily 7 --keep-monthly 6 --prune
restic forget --host myserver --tag databases --keep-daily 14 --keep-monthly 12 --prune

Tags for Organization

Tag backups at creation time to enable scoped listing and retention:

1
2

restic backup --tag files --tag daily /home /etc
restic backup --tag databases --tag daily --stdin --stdin-filename postgres.sql < <(pg_dump mydb)

Integrity Checking

1
2
3
4
5
6
7
8

# Check repository structure (metadata only, fast)
restic check

# Verify actual data blocks — bandwidth-conscious subset
restic check --read-data-subset=10%

# Full data verification (slow, use weekly or monthly)
restic check --read-data

A passing restic check means the repository metadata is consistent. --read-data-subset=10% downloads and verifies a random 10% of pack files, giving you statistical confidence that the data itself is intact without consuming the full bandwidth of a complete verification.

Part 4: Backblaze B2 — Best-Value Cloud Storage for Backups

Backblaze B2 is S3-compatible object storage at a fraction of the cost of AWS S3.

Cost Comparison

B2 egress to the Cloudflare network is free (Bandwidth Alliance). If your restore traffic routes through Cloudflare, you pay nothing for downloads. This is a meaningful cost difference for large restores.

Creating a B2 Bucket

Via the web console:

1. Log in to backblaze.com and navigate to B2 Cloud Storage
2. Create Bucket: choose a globally unique name, set Private (never public for backup data)
3. Enable Object Lock if you want immutable backups (cannot be enabled after creation)
4. Enable versioning (required for Object Lock)

Via the B2 CLI:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

pip install b2
b2 authorize-account  # prompts for application key ID and key

b2 create-bucket \
  --defaultServerSideEncryption SSE-B2 \
  my-backup-bucket allPrivate

# Enable Object Lock (must be done at creation, not after)
b2 create-bucket \
  --defaultServerSideEncryption SSE-B2 \
  --fileLockEnabled \
  my-backup-bucket-locked allPrivate

Application Keys: Principle of Least Privilege

Create a bucket-specific application key for each backup host — not your master key. If a server is compromised, the attacker can only reach that bucket.

In the B2 console: App Keys → Add a New Application Key:

• Name: myserver-backup-key
• Buckets: Select your backup bucket only
• Capabilities: readFiles, writeFiles, deleteFiles, listFiles, listBuckets (Restic needs list and write; restrict delete if using Object Lock)
• File name prefix: optional, useful for multi-tenant buckets

With Object Lock in compliance mode, you can safely grant deleteFiles to the backup key — the lock prevents actual deletion within the retention period.

Connecting Restic to B2

B2 is S3-compatible. Use the S3 backend with B2’s S3-compatible endpoint:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Find your bucket's endpoint in the B2 console
# Format: s3.<region>.backblazeb2.com
# Example: s3.us-west-004.backblazeb2.com

export RESTIC_REPOSITORY="s3:s3.us-west-004.backblazeb2.com/your-bucket-name"
export AWS_ACCESS_KEY_ID="your-b2-key-id"
export AWS_SECRET_ACCESS_KEY="your-b2-application-key"

restic init
restic backup /data

The region-specific endpoint matters — using the wrong one will fail or route traffic inefficiently. Find the correct endpoint for your bucket in the B2 console under Bucket Details.

Monitoring B2 Usage

1
2
3
4
5
6
7
8
9

# Account-level usage
b2 get-account-info

# Bucket size and file count
b2 get-bucket my-backup-bucket
b2 ls --long --recursive b2://my-backup-bucket | tail -1

# List recent uploads
b2 ls --long b2://my-backup-bucket restic/

Part 5: rclone — The Swiss Army Knife of Cloud Storage

rclone is a command-line tool that supports over 70 cloud storage providers with a consistent interface. Where Restic specializes in encrypted deduplicated backups, rclone specializes in raw file synchronization and cloud-to-cloud transfers.

In a complete backup strategy, rclone fills the gaps Restic doesn’t:

• Syncing a raw directory to B2 (no encryption or dedup needed — e.g., already-encrypted archives)
• Mirroring between cloud providers (B2 → Wasabi for geographic redundancy)
• Acting as a transport layer for Restic when the backend is not natively supported
• Mounting cloud storage as a local filesystem

Installation and Configuration

1
2
3
4
5
6
7
8

# Linux
curl https://rclone.org/install.sh | sudo bash

# macOS
brew install rclone

# Configure interactively
rclone config

The rclone config wizard walks you through adding a remote. For Backblaze B2:

Type: Backblaze B2
Account: your-account-id
Key: your-application-key

This creates an entry in ~/.config/rclone/rclone.conf (or /root/.config/rclone/rclone.conf for root). Protect this file:

1

chmod 600 ~/.config/rclone/rclone.conf

The config file contains your storage credentials in plaintext. Consider using rclone config password to encrypt it with a password if the host may be compromised.

Common Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# Copy files to B2 (does not delete destination-only files)
rclone copy /local/path b2:my-bucket/path --progress

# Sync (mirrors source to destination — deletes files not in source)
rclone sync /local/path b2:my-bucket/path --progress

# Sync with versioned archive of deleted files (safer than plain sync)
rclone sync /local/path b2:my-bucket/path \
  --backup-dir b2:my-bucket-archive/$(date +%Y%m%d) \
  --progress

# Verify checksums between local and remote
rclone check /local/path b2:my-bucket/path

# List files
rclone ls b2:my-bucket
rclone lsd b2:my-bucket          # directories only
rclone lsf b2:my-bucket --format "sp"  # size and path

# Bucket usage
rclone size b2:my-bucket

# Mount B2 as a local filesystem
rclone mount b2:my-bucket /mnt/b2 --daemon --vfs-cache-mode writes

rclone for Multi-Cloud Redundancy

Mirror your B2 bucket to a second provider (Wasabi, S3, another B2 region) for geographic redundancy:

1
2
3
4
5
6
7

# Add Wasabi as a remote in rclone config, then:
rclone sync b2:my-backup-bucket wasabi:my-backup-mirror \
  --transfers 4 \
  --checkers 8 \
  --bwlimit 50M \
  --log-file /var/log/rclone-mirror.log \
  --log-level INFO

This gives you a second cloud copy without any additional complexity in your Restic workflow. Run it weekly or nightly after your Restic backup completes.

rclone crypt: Encrypting Any Provider

If you need to use a provider that Restic doesn’t support natively, or you want a second layer of encryption, rclone’s crypt remote encrypts files transparently:

1
2
3
4
5
6

rclone config
# Add new remote, type: crypt
# Remote: b2:my-bucket/encrypted
# Password: your-encryption-password
# Salt: your-password-salt (optional but recommended)
# Name it: b2-encrypted

Now use b2-encrypted: like any other remote — rclone encrypts before upload, decrypts on download. You can then point Restic at this crypt remote:

1

export RESTIC_REPOSITORY="rclone:b2-encrypted:restic-repo"

Key Flags

Part 6: Automated Backup Scripts

The backup only runs if it’s automated. Here are production-ready scripts you can drop in and adapt.

The Core Backup Script

Save as /usr/local/bin/backup.sh, chmod 700, owned by root.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159

#!/usr/bin/env bash
# /usr/local/bin/backup.sh
# Production Restic backup script
# Designed to run via systemd timer or cron

set -euo pipefail

# ── Configuration ─────────────────────────────────────────────────────────────
ENV_FILE="/etc/restic/env"
LOG_FILE="/var/log/restic/backup.log"
LOCK_FILE="/var/run/restic-backup.lock"
HEALTHCHECK_URL="${HEALTHCHECK_URL:-}"          # set in env file
SLACK_WEBHOOK_URL="${SLACK_WEBHOOK_URL:-}"      # set in env file
HOSTNAME="${HOSTNAME:-$(hostname -s)}"

# Retention policy
KEEP_LAST=3
KEEP_DAILY=7
KEEP_WEEKLY=4
KEEP_MONTHLY=12
KEEP_YEARLY=3

# ── Helpers ───────────────────────────────────────────────────────────────────
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

die() {
    log "ERROR: $*"
    notify_failure "$*"
    rm -f "$LOCK_FILE"
    exit 1
}

notify_failure() {
    local message="$1"
    if [[ -n "$SLACK_WEBHOOK_URL" ]]; then
        curl -fsS -X POST "$SLACK_WEBHOOK_URL" \
            -H 'Content-Type: application/json' \
            -d "{\"text\": \":x: *Backup failed on ${HOSTNAME}*\n\`\`\`${message}\`\`\`\"}" \
            || true
    fi
}

ping_healthcheck() {
    local status="$1"   # "" for success, "/fail" for failure, "/start" for start
    if [[ -n "$HEALTHCHECK_URL" ]]; then
        curl -fsS --retry 3 --max-time 10 "${HEALTHCHECK_URL}${status}" || true
    fi
}

# ── Pre-flight ─────────────────────────────────────────────────────────────────
mkdir -p "$(dirname "$LOG_FILE")"
mkdir -p "$(dirname "$LOCK_FILE")"

# Prevent concurrent runs
if [[ -f "$LOCK_FILE" ]]; then
    die "Lock file exists: $LOCK_FILE — is another backup running? PID: $(cat "$LOCK_FILE")"
fi
echo $$ > "$LOCK_FILE"
trap 'rm -f "$LOCK_FILE"' EXIT

# Load credentials
[[ -f "$ENV_FILE" ]] || die "Environment file not found: $ENV_FILE"
# shellcheck source=/etc/restic/env
source "$ENV_FILE"

[[ -n "${RESTIC_REPOSITORY:-}" ]] || die "RESTIC_REPOSITORY not set"
[[ -n "${RESTIC_PASSWORD:-}" ]]   || die "RESTIC_PASSWORD not set"

ping_healthcheck "/start"
log "========== Backup starting on ${HOSTNAME} =========="

# ── File backups ──────────────────────────────────────────────────────────────
log "Backing up: /home /etc /root"
restic backup \
    --tag files \
    --tag "${HOSTNAME}" \
    --host "${HOSTNAME}" \
    --exclude-file=/etc/restic/excludes \
    --exclude='/home/*/.cache' \
    --exclude='/home/*/.local/share/Trash' \
    --exclude='/home/*/node_modules' \
    --exclude='/home/*/go/pkg' \
    --exclude='/home/*/snap' \
    --exclude='/root/.cache' \
    /home /etc /root \
    2>&1 | tee -a "$LOG_FILE" || die "File backup failed"

log "Backing up: /var/lib/docker/volumes"
if [[ -d /var/lib/docker/volumes ]]; then
    restic backup \
        --tag docker-volumes \
        --tag "${HOSTNAME}" \
        --host "${HOSTNAME}" \
        /var/lib/docker/volumes \
        2>&1 | tee -a "$LOG_FILE" || die "Docker volumes backup failed"
fi

# ── Database backups ───────────────────────────────────────────────────────────
# PostgreSQL
if command -v psql &>/dev/null && pg_isready -q 2>/dev/null; then
    log "Backing up PostgreSQL databases"
    databases=$(sudo -u postgres psql -t -c "SELECT datname FROM pg_database WHERE datistemplate = false AND datname != 'postgres';")
    for db in $databases; do
        db="$(echo "$db" | tr -d ' ')"
        log "  Dumping: $db"
        sudo -u postgres pg_dump --format=custom "$db" \
            | restic backup \
                --tag databases \
                --tag postgresql \
                --tag "${HOSTNAME}" \
                --host "${HOSTNAME}" \
                --stdin \
                --stdin-filename "postgres-${db}.dump" \
            2>&1 | tee -a "$LOG_FILE" || die "PostgreSQL backup failed for $db"
    done
fi

# MySQL / MariaDB
if command -v mysqldump &>/dev/null && mysqladmin ping -s 2>/dev/null; then
    log "Backing up MySQL/MariaDB databases"
    databases=$(mysql -N -e "SHOW DATABASES;" | grep -Ev '^(information_schema|performance_schema|sys)$')
    for db in $databases; do
        log "  Dumping: $db"
        mysqldump --single-transaction --routines --events "$db" \
            | restic backup \
                --tag databases \
                --tag mysql \
                --tag "${HOSTNAME}" \
                --host "${HOSTNAME}" \
                --stdin \
                --stdin-filename "mysql-${db}.sql" \
            2>&1 | tee -a "$LOG_FILE" || die "MySQL backup failed for $db"
    done
fi

# ── Retention + Pruning ────────────────────────────────────────────────────────
log "Applying retention policy"
restic forget \
    --host "${HOSTNAME}" \
    --keep-last "${KEEP_LAST}" \
    --keep-daily "${KEEP_DAILY}" \
    --keep-weekly "${KEEP_WEEKLY}" \
    --keep-monthly "${KEEP_MONTHLY}" \
    --keep-yearly "${KEEP_YEARLY}" \
    --prune \
    2>&1 | tee -a "$LOG_FILE" || die "Forget/prune failed"

# ── Integrity check (weekly, on Sunday) ───────────────────────────────────────
if [[ "$(date +%u)" == "7" ]]; then
    log "Running weekly integrity check (10% data subset)"
    restic check --read-data-subset=10% \
        2>&1 | tee -a "$LOG_FILE" || die "Repository integrity check failed"
fi

# ── Success ────────────────────────────────────────────────────────────────────
log "========== Backup completed successfully =========="
ping_healthcheck ""  # ping success

The Excludes File

# /etc/restic/excludes
*.tmp
*.log
*.pid
*.sock
*.swp
*~
.DS_Store
Thumbs.db
lost+found
proc
sys
dev
run
tmp
.Trash
.trash
Cache
cache
.cache
node_modules
.venv
__pycache__
*.pyc
.gradle
.m2
target
build
dist
.terraform
*.tfstate.backup

Systemd Timer (Preferred Over Cron)

Create the service unit at /etc/systemd/system/backup.service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

[Unit]
Description=Restic backup to Backblaze B2
After=network-online.target
Wants=network-online.target
# Don't run if we're on battery (laptops)
ConditionACPower=true

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
# Load environment file for credentials
EnvironmentFile=-/etc/restic/env
# Security hardening
PrivateTmp=true
ProtectSystem=full
NoNewPrivileges=true
# Keep logs in the journal
StandardOutput=journal
StandardError=journal
# Allow the backup to take up to 6 hours
TimeoutStartSec=6h

Create the timer unit at /etc/systemd/system/backup.timer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

[Unit]
Description=Run Restic backup daily
Requires=backup.service

[Timer]
# Run at 2:00am daily
OnCalendar=*-*-* 02:00:00
# Add up to 30 minutes of random delay to spread backup load
RandomizedDelaySec=1800
# Run immediately if we missed the last scheduled run (e.g., system was off)
Persistent=true

[Install]
WantedBy=timers.target

Enable and start:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

systemctl daemon-reload
systemctl enable --now backup.timer

# Verify
systemctl list-timers backup.timer
systemctl status backup.timer

# Trigger a manual run
systemctl start backup.service
journalctl -u backup.service -f

Cron Alternative

If you prefer cron or are on a system without systemd:

1
2
3

# /etc/cron.d/restic-backup
# Run at 2:15am daily (offset to avoid busy cron time)
15 2 * * * root /usr/local/bin/backup.sh >> /var/log/restic/backup.log 2>&1

Part 7: Healthchecks — Knowing Your Backups Actually Ran

Automated backups fail silently. The job exits with an error, cron doesn’t email you because nobody configured mail, and you don’t notice for six months — until you need the backup.

The dead man’s switch pattern solves this: the backup pings a monitoring service on success. If the ping doesn’t arrive within the expected window, the monitoring service alerts you. Silence is the failure signal.

Healthchecks.io

Healthchecks.io is the canonical service for this pattern. Free tier allows 20 checks with 1-month log retention. It’s also open-source and self-hostable if you prefer.

Setup:

1. Create an account, click New Check
2. Set schedule: Daily, Grace period: 1 hour (buffer for backup duration variance)
3. Copy the ping URL: https://hc-ping.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
4. Add to /etc/restic/env: export HEALTHCHECK_URL="https://hc-ping.com/your-uuid"

The backup script above uses ping_healthcheck "/start" at the beginning and ping_healthcheck "" on success. Healthchecks.io records both, letting you see backup duration over time.

For failures, the script calls ping_healthcheck "/fail" which marks the check as failed immediately rather than waiting for the deadline.

Self-Hosted Alternative: Uptime Kuma

Uptime Kuma supports push monitoring (the reverse of its primary use case). Add a push monitor, copy the URL, use it exactly like the healthchecks.io URL. Combine with its notification integrations for Slack, Discord, email, Telegram, and more.

What to Monitor

Beyond the alive/dead signal:

• Last successful backup timestamp: alert if more than 25 hours old
• Backup size trends: a sudden large increase may indicate a misconfiguration backing up unwanted data; a sudden decrease may indicate files went missing
• Repository check results: alert immediately if restic check exits non-zero
• Storage cost trends: unusual cost spikes indicate something unexpected is being uploaded

Part 8: Testing Restores — The Step Everyone Skips

Every sysadmin who has ever been paged during a crisis because of a backup that “definitely should have worked” will tell you the same thing: test your restores on a schedule, not at 3am.

Types of Restore Tests

File restore test (weekly automation): Restore a specific file from the latest snapshot and compare its SHA-256 checksum against the live copy. This verifies the repository is readable and the data is intact.

Application restore test (monthly, manual): Restore a database dump to a staging instance and verify the application starts and data looks correct. This catches issues that a file checksum doesn’t surface — a dump that restores but contains corrupt data, a missing migration, a dependency on a config file that wasn’t included in the backup.

Full restore test (quarterly): Provision a new VM or server, restore everything from the backup, and verify all services come up correctly. Time this — that duration is your actual RTO. Document the procedure so that whoever is on-call during a real disaster has a step-by-step runbook.

Automated File Restore Test Script

Save as /usr/local/bin/test-restore.sh, add to a weekly systemd timer or cron:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

#!/usr/bin/env bash
# /usr/local/bin/test-restore.sh
# Weekly automated restore verification

set -euo pipefail

ENV_FILE="/etc/restic/env"
LOG_FILE="/var/log/restic/restore-test.log"
RESTORE_DIR="/tmp/restic-restore-test-$$"
HEALTHCHECK_URL="${RESTORE_HEALTHCHECK_URL:-}"  # separate check from main backup
SLACK_WEBHOOK_URL="${SLACK_WEBHOOK_URL:-}"

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }

cleanup() {
    rm -rf "$RESTORE_DIR"
    log "Cleaned up restore directory"
}
trap cleanup EXIT

source "$ENV_FILE"

log "========== Restore test starting =========="
mkdir -p "$LOG_FILE" 2>/dev/null || true
mkdir -p "$(dirname "$LOG_FILE")"

# Pick a test file — /etc/passwd is always present and doesn't change often
# In production, choose a file representative of what you care about
TEST_FILE="/etc/passwd"
ORIGINAL_CHECKSUM=$(sha256sum "$TEST_FILE" | awk '{print $1}')

log "Test file: $TEST_FILE"
log "Original SHA256: $ORIGINAL_CHECKSUM"

# Get the latest snapshot ID
LATEST_SNAPSHOT=$(restic snapshots --json --latest 1 | jq -r '.[0].id')
log "Latest snapshot: $LATEST_SNAPSHOT"
log "Snapshot time: $(restic snapshots --json --latest 1 | jq -r '.[0].time')"

# Restore the specific file
mkdir -p "$RESTORE_DIR"
log "Restoring to: $RESTORE_DIR"

restic restore "$LATEST_SNAPSHOT" \
    --include "$TEST_FILE" \
    --target "$RESTORE_DIR" \
    2>&1 | tee -a "$LOG_FILE"

RESTORED_FILE="${RESTORE_DIR}${TEST_FILE}"

if [[ ! -f "$RESTORED_FILE" ]]; then
    log "FAIL: Restored file not found at $RESTORED_FILE"
    [[ -n "$SLACK_WEBHOOK_URL" ]] && curl -fsS -X POST "$SLACK_WEBHOOK_URL" \
        -H 'Content-Type: application/json' \
        -d "{\"text\": \":x: *Restore test FAILED on $(hostname)*: file not found after restore\"}" \
        || true
    exit 1
fi

RESTORED_CHECKSUM=$(sha256sum "$RESTORED_FILE" | awk '{print $1}')
log "Restored SHA256: $RESTORED_CHECKSUM"

if [[ "$ORIGINAL_CHECKSUM" == "$RESTORED_CHECKSUM" ]]; then
    log "PASS: Checksums match"
    [[ -n "$HEALTHCHECK_URL" ]] && curl -fsS --retry 3 "$HEALTHCHECK_URL" || true
    log "========== Restore test PASSED =========="
else
    log "FAIL: Checksum mismatch — original: $ORIGINAL_CHECKSUM, restored: $RESTORED_CHECKSUM"
    [[ -n "$SLACK_WEBHOOK_URL" ]] && curl -fsS -X POST "$SLACK_WEBHOOK_URL" \
        -H 'Content-Type: application/json' \
        -d "{\"text\": \":x: *Restore test FAILED on $(hostname)*: checksum mismatch for ${TEST_FILE}\"}" \
        || true
    exit 1
fi

Add a weekly systemd timer for this script using the same pattern as the backup timer, but scheduled for Sunday morning after the integrity check.

Running a Repository Integrity Check

1
2
3
4
5
6
7
8

# Check 5% of data blocks (fast, ~weekly)
restic check --read-data-subset=5%

# Check 25% of data blocks (monthly)
restic check --read-data-subset=25%

# Full check (quarterly, or before a planned migration)
restic check --read-data

Measuring Your Actual RTO

When you run your quarterly full restore test, time it explicitly:

1

time restic restore latest --target /mnt/restore-test

Also time:

• Database import time
• Application startup and health check passing
• DNS propagation (if restoring to a new IP)

Document the total time. That is your actual RTO. If it exceeds your recovery objective, you have work to do — either faster storage, a parallel restore process, or a pre-provisioned warm standby.

Chaos Backup Testing

Once a year, deliberately corrupt a non-production environment and run through a full recovery. The goal is not to verify that the backup software works — it’s to verify that your runbook is complete, your credentials are accessible, and the recovery procedure works when you’re stressed and it’s the middle of the night.

Document every step that required guessing or looking something up. Those gaps are documentation debt. Fill them before the next real incident.

Part 9: What to Back Up (and What Not To)

Backing up everything wastes storage and makes restores slower. Backing up too little means you can’t recover. Here is the practical breakdown.

Back Up

• Home directories: user configs, documents, projects, SSH keys
• Application data: /var/lib, service-specific data directories
• Database dumps: always dump databases separately and pipe to Restic — don’t try to back up raw database files with the database running
• Configuration files: /etc, application configs, .env files, docker-compose files
• Docker named volumes: restic backup /var/lib/docker/volumes
• TLS certificates and private keys: especially if using custom PKI
• Secrets and credentials: encrypted, ideally via a secrets manager that has its own backup
• Crontabs and systemd units: the automation config that runs your systems

Don’t Back Up

• Operating system files: reinstall from scratch — it’s faster and cleaner than restoring an OS
• node_modules, .venv, .gradle, target/, build/, dist/ — restore from package manager
• Docker image layers: pull from the registry; back up only the config that defines what to pull
• Temporary files, caches: /tmp, ~/.cache, browser caches
• Your backup destination: don’t include the directory where local backups live in your backup job
• Reproducible build artifacts: anything that can be deterministically regenerated from source and a lock file

.resticignore Example (Server)

# /etc/restic/excludes — server configuration

# Package manager caches
/var/cache/apt
/var/cache/dnf
/var/cache/yum

# Logs (use a log shipper instead)
/var/log
/home/*/.pm2/logs

# Containers
/var/lib/docker/overlay2
/var/lib/docker/image
/var/lib/containerd

# Temporary
/tmp
/var/tmp
/run

# Already-offsite data
/mnt/b2

# Build artifacts
**/node_modules
**/.venv
**/target
**/dist
**/build
**/__pycache__
**/*.pyc

.resticignore Example (Developer Workstation)

# ~/.config/restic/excludes — developer workstation

# Package and build caches
node_modules
.venv
env
venv
.gradle
.m2
.ivy2
target
dist
build
__pycache__
*.pyc
*.pyo
.mypy_cache
.pytest_cache
.ruff_cache

# IDEs
.idea
*.iml
.vscode/extensions

# macOS
.DS_Store
.Spotlight-V100
.Trashes
.TemporaryItems

# Large media (back up separately or accept the loss)
~/Downloads
~/Movies

# Browser caches
*/Chrome/*/Cache
*/Firefox/*/Cache
*/Safari/Caches

# Vagrant, Docker local volumes not worth backing up
.vagrant

Part 10: Backup Strategy by System Type

Linux Server (VPS or Bare Metal)

The standard setup described throughout this guide:

• systemd timer running backup.sh at 2am
• Restic to B2 with daily/weekly/monthly retention
• Database dumps piped directly to Restic stdin
• Weekly restic check --read-data-subset=10%
• Healthchecks.io for monitoring

Encryption means your VPS provider can’t read your backups. Deduplication means the second backup of a largely-unchanged system uploads in seconds.

NAS (Synology / QNAP)

Synology:

• Hyper Backup is the native solution — solid, supports B2 as a destination
• For Restic: install via Synology’s package manager or run in a Docker container
• rclone sync is excellent for bulk media that doesn’t need encryption or versioning

QNAP:

• Hybrid Backup Sync supports B2 natively
• Restic available as a QPKG or Docker container

For a NAS, consider a cross-backup between two NAS units at different physical locations — each backs up to the other over WireGuard VPN, plus each backs up to B2. That gives you three copies in three locations without relying entirely on cloud.

Proxmox VMs and LXC Containers

• Proxmox Backup Server (PBS): runs on a separate machine (or VM), handles incremental VM and LXC backups, stores deduplicated snapshots
• vzdump: built-in Proxmox backup, outputs to local storage or NFS
• rclone: rclone sync /mnt/pbs-datastore b2:my-proxmox-backup to push PBS data offsite
• Per-application: inside each VM, run the standard Restic backup in addition to VM-level backups

Don’t rely solely on VM snapshots for application data. A VM snapshot at the hypervisor level does not protect against accidental deletion inside the VM, and restoring an entire VM when you only need one database table is wasteful.

Docker Compose Stacks

The key insight: your Docker Compose config is code — keep it in git. Only the named volumes contain state that isn’t reproducible.

1
2
3
4

# Back up named volumes by stopping the container or using a snapshot-capable filesystem
docker compose down
restic backup /var/lib/docker/volumes/mystack_postgres_data
docker compose up -d

For services that can take a hot backup:

1
2
3
4
5
6
7
8
9

# PostgreSQL inside a container
docker exec my-postgres pg_dumpall -U postgres \
  | restic backup --stdin --stdin-filename postgres-all.sql

# Redis with RDB persistence
docker exec my-redis redis-cli BGSAVE
docker cp my-redis:/data/dump.rdb /tmp/
restic backup /tmp/dump.rdb
rm /tmp/dump.rdb

The compose files themselves — back them up in a git repository and push to a remote (GitHub, Gitea, etc). That copy is separate from your Restic backup chain and gives you immediate access to service definitions without restoring anything.

macOS Workstation

• Time Machine: local backup to an external drive or Time Capsule — fast, easy, built-in
• Restic to B2: offsite copy of the things Time Machine has but that can’t travel home with you

1
2
3
4
5
6
7
8

# Run from launchd (macOS systemd equivalent)
# ~/Library/LaunchAgents/com.user.restic-backup.plist
restic backup \
  --exclude-file="$HOME/.config/restic/excludes" \
  "$HOME/Documents" \
  "$HOME/Projects" \
  "$HOME/.ssh" \
  "$HOME/.config"

Time Machine handles the “restore my entire Mac from last Tuesday” scenario. Restic handles the “my laptop was stolen and I need my SSH keys and project code on a new machine in 30 minutes” scenario.

Kubernetes

• Velero: the standard tool for Kubernetes cluster backup — backs up cluster state (all resources) and persistent volume data to S3/B2
• Application-level backups: still needed — database dumps from pods, application-specific export tools
• etcd backup: critical for control plane recovery, separate from Velero

Kubernetes adds complexity: you need to back up both the cluster state (what should be running) and the data (what your applications have stored). Velero handles the cluster state. Application-level backups handle the data.

Putting It All Together

A complete backup strategy for a Linux homelab or production server, using everything above:

Start simple: Restic to B2, a healthcheck, and a weekly file restore test. That alone puts you ahead of 90% of setups. Add the rest as your comfort and infrastructure complexity grow.

The emotional reality is that most people configure their first real backup strategy after losing data. Don’t be that person. Spend the two hours now. Run restic restore latest --target /tmp/test and verify what comes back. Then sleep better knowing that when something breaks — and it will — you have a tested path to recovery.

Further reading: Restic documentation — the official docs are unusually good. Backblaze B2 S3-compatible API — covers every endpoint and authentication detail. rclone documentation — comprehensive guides for every supported provider.