Proxmox Backup Server In Depth
Proxmox Backup Server In Depth
Proxmox Backup Server (PBS) is a dedicated backup solution designed specifically for Proxmox VE environments. Unlike generic backup tools bolted onto a hypervisor, PBS was built from the ground up around the specific characteristics of VM and container backups: large, structured disk images that change incrementally between snapshots.
The result is a backup system with genuinely impressive efficiency: incremental forever backups, client-side deduplication across all backups in a datastore, optional end-to-end encryption, and a verification system that continuously validates backup integrity. This post covers how it works internally, how to operate it at scale, and how to build a resilient multi-site backup architecture.
Architecture Overview
Core Components
Datastore: the primary organizational unit in PBS. A datastore is a directory on the PBS filesystem where backup data is stored. Each datastore maintains its own chunk store, catalog, and retention configuration. You might have separate datastores for different environments (production, staging) or different retention requirements.
Chunk Store: the heart of PBS’s efficiency. When PBS stores a backup, it breaks the disk image into fixed-size chunks (4 MB by default), computes a SHA-256 hash of each chunk, and stores only unique chunks. If the same 4 MB block exists in ten different backups across ten different VMs, it’s stored exactly once. This deduplication operates across all backups in a datastore — not just multiple backups of the same VM.
Backup Groups: backups are organized as type/name — for example vm/100 or ct/200. Within a group, each backup is a snapshot identified by a timestamp.
Catalog: a fast lookup index that allows PBS to list backup contents and restore individual files without loading the full backup.
Backup Types
PBS supports three backup types:
- VM backups (
vm/): QEMU virtual machine disk images - Container backups (
ct/): LXC container filesystems - Host backups (
host/): arbitrary directories from the PBS client — used for backing up non-Proxmox hosts with theproxmox-backup-clienttool
How Incremental Backups Work
PBS uses a “dirty bitmap” approach for VM backups. QEMU maintains a dirty bitmap that tracks which 64 KB blocks have changed since the last backup. On the next backup run:
- PBS reads the dirty bitmap from QEMU
- Only reads the changed 64 KB regions from the disk
- Rechunks those regions into 4 MB chunks
- Deduplicates against the chunk store (most chunks will already exist)
- Writes only new unique chunks
The result: after the first full backup, subsequent backups typically transfer 1–5% of the disk size and complete in minutes regardless of disk size. A 500 GB VM with 2 GB of changes since the last backup takes roughly the same time to back up as a 50 GB VM with 2 GB of changes.
Installation and Initial Setup
Installing PBS
PBS is a separate Debian-based distribution (not installed on Proxmox VE nodes). Download the ISO from proxmox.com/downloads.
Install on dedicated hardware (or a VM on a separate host — don’t run PBS on the same host whose VMs it’s backing up). Minimum recommended specs:
- 4 CPU cores
- 8 GB RAM (16+ GB for large datastores)
- Boot drive: 32 GB SSD (OS only)
- Backup storage: sized to your backup data × dedup ratio × retention count
Initial Configuration
After installation, access the web UI at https://pbs-host:8007.
|
|
Creating a Datastore
|
|
Adding PBS to Proxmox VE
In each Proxmox VE cluster, add the PBS instance as a storage target:
|
|
Or via the PVE web UI: Datacenter → Storage → Add → Proxmox Backup Server.
Deduplication Internals
Understanding PBS’s chunk store helps you make good decisions about datastore design and hardware sizing.
The Chunk Store on Disk
|
|
What Determines Dedup Ratio?
Good dedup ratios (3–10×) come from:
- Many similar VMs: if you have 20 web servers built from the same template, their base OS is deduplicated across all of them
- Frequent backups: more backup snapshots means more opportunities to find duplicate chunks
- Large, compressible data: databases, logs, and document files compress and dedup well
- Stable data: files that don’t change between backups contribute heavily to dedup savings
Poor dedup ratios (<2×) come from:
- Encrypted VM disks: already-encrypted data has high entropy and doesn’t compress or dedup
- Databases with random I/O patterns: write-heavy databases scatter changes across large files
- Video/media files: already compressed, very low dedup potential
Chunk Store Garbage Collection
When backups are deleted (by retention policy or manually), the chunks they referenced may become unreferenced if no other backup uses them. Garbage collection reclaims this space:
|
|
GC is a two-phase process:
- Mark phase: scan all backup manifests to find all referenced chunk hashes
- Sweep phase: delete any chunk files not referenced by any backup
GC can be I/O intensive on large datastores. Schedule it during off-peak hours.
Encryption
PBS supports end-to-end client-side encryption. Encryption happens on the Proxmox VE host before data is sent to PBS — the PBS server never sees unencrypted data.
Encryption Architecture
PBS uses AES-256-GCM for chunk encryption. The encryption key is generated on the client, and chunks are encrypted before being sent to the server. The key can be protected with a passphrase.
Important implications:
- PBS server compromise doesn’t expose your data
- You cannot recover backups without the encryption key — store it safely, separately from PBS
- Deduplication still works across encrypted backups from the same client (same key = same encrypted chunk for same data)
- Deduplication does NOT work across different encryption keys
Setting Up Encryption
|
|
Configure encryption in PVE backup jobs:
|
|
Key Management
The encryption key file must be available on the PVE host when backups run. For automated backups, remove the passphrase or store the passphrase in a secrets manager:
|
|
Retention Policies
Retention policies determine how many backup snapshots to keep. PBS implements a “keep” policy that’s evaluated per backup group, typically expressed as:
keep-last=N keep the N most recent backups
keep-hourly=N keep the most recent backup for each of the last N hours
keep-daily=N keep the most recent backup for each of the last N days
keep-weekly=N keep the most recent backup for each of the last N weeks
keep-monthly=N keep the most recent backup for each of the last N months
keep-yearly=N keep the most recent backup for each of the last N years
These combine. Proxmox evaluates which backups to keep by applying all rules and keeping any backup that satisfies at least one rule.
Configuring Retention
|
|
In PVE backup jobs (via web UI: Datacenter → Backup → Add/Edit):
keep-last: 3
keep-daily: 14
keep-weekly: 8
keep-monthly: 6
keep-yearly: 2
Example of what this keeps for a daily backup:
- The 3 most recent backups (last 3 days)
- One backup per day for the last 14 days
- One backup per week for the last 8 weeks (~2 months)
- One backup per month for the last 6 months
- One backup per year for the last 2 years
This gives good coverage with reasonable storage use. Adjust based on your RPO requirements and storage budget.
Prune vs Garbage Collection
prune marks old backup snapshots for deletion according to the retention policy. gc (garbage collection) actually reclaims the disk space by removing unreferenced chunks. Both steps are needed for space to be freed:
|
|
Tape Support
PBS supports writing backups to tape via the Linux tape subsystem (LTO drives, tape libraries). Tape is ideal for long-term archival and air-gapped offsite storage.
Tape Hardware Setup
|
|
Tape Pools and Media
|
|
Tape Backup Jobs
|
|
PBS writes backups from the disk datastore to tape in a format that supports direct restore — you don’t need to stage back to disk before restoring.
Replication Between PBS Instances
For true offsite backup protection, replicate your PBS datastore to a remote PBS instance. This is the “3rd copy” in a 3-2-1 backup strategy.
Setting Up Sync Jobs
|
|
The sync job is incremental — it only transfers chunks that don’t already exist on the remote instance. After the initial full sync, daily syncs transfer only new or changed chunks, typically a small fraction of the total datastore size.
Bandwidth Limiting
|
|
Verifying Remote Copies
|
|
Backup Verification
PBS continuously verifies backup integrity. This is crucial — a backup that can’t be restored is not a backup.
Verification Mechanics
PBS verification:
- Reads every chunk referenced by the backup manifests
- Verifies the SHA-256 hash of each chunk matches the chunk’s filename
- Optionally verifies that the backup can be fully reconstructed (checks manifest completeness)
|
|
Test Restores
Verification confirms data integrity but not restorability. Periodically test full restores:
|
|
Automate this process quarterly or whenever backup software changes.
Scheduling Backup Jobs
Backup Job Configuration in PVE
|
|
Backup modes:
snapshot: uses QEMU dirty bitmaps for incremental backup — fastest, recommendedsuspend: suspends the VM briefly to take a consistent snapshot — for VMs that don’t support live snapshotsstop: stops the VM, backs up, restarts — slowest, most consistent
Staggering Backup Jobs
Don’t back up all VMs simultaneously. Stagger start times to avoid I/O saturation:
|
|
Restoring at Scale
Single VM Restore
|
|
File-Level Restore
PBS can mount a VM backup as a FUSE filesystem, allowing individual file recovery without restoring the entire disk:
|
|
Bulk Restore (Disaster Recovery)
When restoring an entire environment:
|
|
For DR scenarios where the original PVE cluster is gone, you can restore directly from a standalone PBS instance without needing PVE infrastructure:
|
|
Monitoring PBS
Built-in Dashboard
PBS’s web UI shows:
- Datastore usage and dedup ratios
- Recent backup task status (success/failure)
- Verification job status
- GC and prune job history
Prometheus Integration
|
|
Key metrics to alert on:
|
|
Backup Job Notification
PBS and PVE can send email notifications on backup completion/failure:
|
|
Sizing PBS Storage
Estimating Storage Requirements
Required raw storage = (total VM disk size × change rate × backup frequency × retention days) / dedup ratio
Example:
- 20 VMs averaging 100 GB each = 2 TB total disk
- 5% daily change rate = 100 GB changes/day
- Daily backups retained for 30 days
- Estimated dedup ratio: 4×
Storage = (2000 GB initial + 100 GB × 30 days) / 4 = (2000 + 3000) / 4 = 1.25 TB
Add 30% headroom: 1.25 × 1.3 = ~1.6 TB
The dedup ratio is the wildcard. Measure your actual ratio after a week of backups and adjust capacity planning accordingly.
Storage Recommendations
PBS benefits from fast storage for the chunk store metadata (small random reads/writes) and sequential throughput for chunk data:
- Boot/OS: 32 GB SSD
- Chunk store: ZFS on spinning HDDs works well (ZFS ARC caches hot chunks in RAM). NVMe or SATA SSD dramatically speeds up verification and restore.
- RAM: 1 GB per TB of backup data for ZFS ARC (8 GB minimum). ZFS ARC is critical for dedup performance.
For the PBS server itself, ZFS is strongly recommended:
|
|
ZFS compression (zstd) provides a second layer of space savings on top of PBS’s deduplication.
Production Checklist
Setup
- PBS installed on dedicated hardware (not on PVE nodes being backed up)
- Separate PBS instance at an offsite location for 3-2-1 compliance
- ZFS on backup storage with appropriate RAIDZ or mirror level
- TLS fingerprint pinned in PVE storage config
Encryption
- Encryption key generated and stored in a password manager or key escrow
- Key backup stored separately from PBS and PVE infrastructure
- Encryption enabled on all backup jobs
Retention
- Retention policy matches RPO requirements
- GC scheduled after prune jobs
- Retention policy documented and approved by stakeholders
Verification
- Automated verification job scheduled weekly
- Quarterly test restore documented and tracked
- Alerts configured for verification failures
Replication
- Sync job to offsite PBS instance configured
- Sync job alerts on failure
- Bandwidth limits set to avoid saturating WAN
Monitoring
- Prometheus metrics scraped from PBS
- Alerts for backup failure, datastore capacity, sync failure
- Email notifications enabled for backup jobs
Proxmox Backup Server is one of the most polished open-source backup solutions available for hypervisor environments. When configured correctly — with encryption, offsite replication, automated verification, and tested restore procedures — it provides enterprise-grade backup protection at a fraction of the cost of commercial alternatives.
Comments