Proxmox VE: The Comprehensive Guide

May 23, 2026 26 min read Homelab & Self-Hosting

proxmoxhomelabvirtualizationlinuxdevopsself-hostedopen-sourcestorage

Contents

Proxmox VE sits in a rare category of software: it is genuinely enterprise-capable, fully open-source, and free to use without a subscription. VMware ESXi served this role for a decade, but its acquisition by Broadcom and subsequent licensing changes sent a significant fraction of the homelab and small-business market looking for an alternative. Proxmox was already the answer for many; it is now the answer for many more.

What makes Proxmox worth understanding in depth is that it is not just a hypervisor. It is a converged platform that manages KVM virtual machines, LXC containers, ZFS or Ceph storage, software-defined networking, clustering, high availability, and backups from a single web interface and a coherent REST API. The learning curve is real — the defaults are not always obvious, and some decisions made at installation (storage layout, network bridge names) are painful to change later. Understanding the platform before you deploy saves significant time.

This guide covers the full operational picture: installation, storage, networking, VM and container management, clustering, HA, GPU passthrough, backup, and automation. It is long because Proxmox is deep.

What Proxmox VE Is

Proxmox VE runs on Debian Linux (currently Debian 12 Bookworm as of PVE 8.x). On top of Debian it adds:

KVM hypervisor via qemu-kvm for full hardware virtualization. Any OS that runs on x86-64 hardware runs as a KVM VM.
LXC (Linux Containers) for lightweight OS-level containerization. LXC containers share the host kernel and start in under a second. They are not Docker containers — they are more like a lightweight VM with a full Linux userspace.
ZFS integrated at the kernel level, available as a storage backend since PVE 3 (2014).
Ceph storage cluster management integrated into the web UI and CLI.
Corosync clustering for multi-node management and high availability.
pvesm (Proxmox VE Storage Manager) abstracting local and network storage backends.
pveam for container template management.
A REST API covering every operation available in the web UI.

The web interface runs on port 8006 over HTTPS. All configuration lives in /etc/pve/, which is distributed across cluster nodes in real time via pmxcfs (a cluster filesystem backed by Corosync).

Installation

Download the ISO from proxmox.com/en/downloads. Burn to USB with dd or Ventoy. Boot the target machine.

The installer prompts for:

Target disk: choose the disk for the OS installation. If you intend to use ZFS for VM storage, do not use your only SSD for the OS — use a separate smaller disk or a mirrored pair of cheap SSDs.
Filesystem: ext4 or zfs (RAID-0, RAID-1, RAID-10, RAIDZ-1/2/3). For a single-node homelab on a single OS disk, ext4 is fine. For production with redundancy requirements, ZFS RAID-1 (mirror) on two OS disks is safer.
Network configuration: IP, gateway, DNS. Choose a static IP on your management network. This is the address the web UI will be on.
Hostname: set a fully qualified hostname (pve1.homelab.local). This matters for clustering — it must be resolvable from other nodes.

Post-install: repository configuration

The default Proxmox installation has the enterprise repository enabled, which requires a paid subscription to use. Without a subscription, apt update fails. Switch to the no-subscription repository:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Disable enterprise repo
cat /etc/apt/sources.list.d/pve-enterprise.list
# comment out or delete the line

# Add no-subscription repo
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" \
  > /etc/apt/sources.list.d/pve-no-subscription.list

# Also disable Ceph enterprise repo if present
cat /dev/null > /etc/apt/sources.list.d/ceph.list

# Update and upgrade
apt update && apt dist-upgrade -y

The no-subscription repository gets the same packages as the enterprise repository — the subscription is for access to the stable enterprise repo and commercial support, not different software.

Post-install: remove subscription nag

The web UI shows a subscription nag on login. Remove it:

1
2
3
4


# On PVE 8.x
sed -Ezi.bak "s/(Ext.Msg.show\(\{[^}]+\}[^}]+\}[^}]+\});/void(0);/g" \
  /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js
systemctl restart pveproxy

Note: this change is overwritten on proxmox-widget-toolkit package updates. Re-apply after upgrades.

Post-install: IOMMU for GPU passthrough

If you plan to use PCI passthrough, enable IOMMU now (before you have VMs to migrate):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# Intel systems — edit GRUB
sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"/' \
  /etc/default/grub
update-grub

# AMD systems
sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT="quiet"/GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"/' \
  /etc/default/grub
update-grub

# Load VFIO modules
echo -e "vfio\nvfio_iommu_type1\nvfio_pci\nvfio_virqfd" >> /etc/modules
update-initramfs -u -k all

reboot

# After reboot, verify IOMMU is active
dmesg | grep -e DMAR -e IOMMU | head -20

iommu=pt (passthrough mode) reduces IOMMU overhead for devices that are not being passed through. Set it regardless of whether you are using passthrough — it improves performance for all VMs.

Storage Backends

Proxmox abstracts storage into storage pools. Each pool has a type, a name, and a set of content types it can hold (disk images, ISO images, container templates, snippets, backups, etc.).

Datacenter → Storage → Add

Local storage (default)

The default installation creates two storage entries:

local (directory at /var/lib/vz/) — holds ISOs, container templates, backups. Type: Directory.
local-lvm (LVM thin pool on the OS disk) — holds VM disk images and container volumes. Type: LVM-Thin.

LVM-Thin provides thin provisioning (disk images are sparse until written) and fast snapshots at the LVM level. It is adequate for a single node with a large OS disk but does not survive node failure.

ZFS storage pools

ZFS is the recommended storage backend for serious homelab use. Add drives to a ZFS pool after installation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Create a mirror pool from two drives
zpool create vmdata mirror /dev/sdb /dev/sdc

# Or RAIDZ2 across six drives
zpool create vmdata raidz2 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg

# Set recommended options for VM workloads
zfs set atime=off vmdata
zfs set compression=lz4 vmdata
zfs set xattr=sa vmdata
zfs set dnodesize=auto vmdata

Add the pool to Proxmox:

Datacenter → Storage → Add → ZFS
  ID: vmdata
  Pool: vmdata
  Thin Provision: checked
  Content: Disk image, Container

Or via CLI:

1

pvesm add zfspool vmdata --pool vmdata --sparse 1 --content images,rootdir

ZFS pools in Proxmox support snapshots natively, which Proxmox uses for VM snapshots and backup. The ARC (ZFS read cache) significantly improves I/O for frequently accessed VM disks — tune zfs_arc_max to leave enough RAM for the VMs themselves.

NFS and SMB/CIFS

For shared storage across multiple nodes (required for live migration of VMs without shared storage), NFS is the common choice:

Datacenter → Storage → Add → NFS
  ID: nas-storage
  Server: 192.168.1.50
  Export: /srv/proxmox
  Content: Disk image, ISO image, Backup, Snippets

NFS storage enables VM migration between nodes — the disk image lives on the NFS server rather than on a specific node. All nodes in the cluster can access it simultaneously.

Ceph

Ceph provides distributed block storage across cluster nodes. Each node contributes disks as OSDs (Object Storage Daemons); Ceph replicates data across them for redundancy. For a Proxmox cluster with three or more nodes each having dedicated storage disks, Ceph is the production-grade shared storage option.

Minimum Ceph cluster for homelab: 3 nodes, 1 OSD per node, at least 1 dedicated Ceph network (10GbE recommended).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# On all cluster nodes — install Ceph
pveceph install --repository no-subscription

# Initialize Ceph on the first node
pveceph init --network 10.10.10.0/24  # dedicated Ceph network

# Create monitors (one per node)
pveceph mon create

# On each node — add OSD (replace /dev/sdd with your Ceph disk)
pveceph osd create /dev/sdd

# Create a pool for VM images (replace 32 with PG count: ~ 100 OSDs × 0.1 × 3 replicas)
pveceph pool create vmpool --pg_num 32 --size 3

# Add to Proxmox storage
pvesm add rbd ceph-vmpool --pool vmpool --content images,rootdir

The web UI at Datacenter → Ceph shows OSD status, pool utilization, and health in real time.

Networking

Default bridge

The installer creates vmbr0 bridged to the first physical NIC. All VMs on the default network connect to vmbr0 as their uplink and get access to the same L2 segment as the host.

The host’s IP address is on vmbr0, not on the physical NIC directly. This is standard Linux bridging — the physical NIC (eno1 or eth0) has no IP, the bridge has the IP, and packets flow through the bridge.

# /etc/network/interfaces
auto eno1
iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
    address 192.168.1.10/24
    gateway 192.168.1.1
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0

VLANs

For multi-tenant or segmented networks, use VLAN-aware bridges. Enable VLAN awareness on vmbr0:

Node → Network → vmbr0 → Edit → VLAN aware: checked

With VLAN awareness enabled, VMs can specify a VLAN tag in their network device configuration. The bridge handles 802.1Q tagging transparently. The physical switch port must be configured as a trunk.

For a dedicated management VLAN (e.g., VLAN 10) and a VM traffic VLAN (e.g., VLAN 20):

# /etc/network/interfaces
auto vmbr0.10
iface vmbr0.10 inet static
    address 192.168.10.10/24
    gateway 192.168.10.1

auto vmbr0.20
iface vmbr0.20 inet manual

VMs on VLAN 20 set their network device tag to 20 and the bridge forwards appropriately.

Bond (NIC teaming)

For redundancy or additional bandwidth on multi-NIC hosts:

Node → Network → Create → Linux Bond
  Name: bond0
  Slaves: eno1 eno2
  Mode: LACP (802.3ad) — requires switch support
     or active-backup — works without switch config

# Then create the bridge on top of bond0
auto vmbr0
iface vmbr0 inet static
    bridge-ports bond0
    ...

LACP requires the switch to be configured with a LAG. active-backup is simpler and provides redundancy without switch configuration — if eno1 fails, traffic moves to eno2 automatically.

Dedicated networks for clustering and storage

For a cluster, separate networks for different traffic types:

Management/VM network: vmbr0 on the main NIC (192.168.1.x)
Corosync/cluster heartbeat: dedicated NIC or VLAN, low latency required
Ceph/storage replication: dedicated NIC, high bandwidth (10GbE)
VM live migration: shared with storage or dedicated

Keeping Corosync traffic off the VM network prevents a traffic spike from causing false node failures.

KVM Virtual Machines

Creating a VM

Upload an ISO to the local storage first:

local → ISO Images → Upload

Then create the VM:

Create VM
  General: Node, VM ID (100+), Name
  OS: ISO image, Guest OS type (Linux 6.x kernel, Windows 11, etc.)
  System:
    Machine: q35 (preferred over i440fx for PCIe support and modern features)
    BIOS: SeaBIOS (legacy) or OVMF (UEFI — required for Secure Boot and some Windows installs)
    SCSI Controller: VirtIO SCSI single (best performance)
    Qemu Agent: checked (install qemu-guest-agent in the guest later)
  Disks:
    Bus: VirtIO Block or SCSI (VirtIO is faster; SCSI supports more features)
    Storage: vmdata (your ZFS pool)
    Size: as needed
    Cache: None (safest with ZFS) or Write Back (faster, less safe)
    Discard: checked (TRIM/unmap support — requires thin-provisioned storage)
    IO thread: checked (per-disk thread improves multi-disk I/O)
  CPU:
    Sockets/Cores: as needed
    Type: host (exposes all host CPU features; not migratable across different CPU generations)
         or x86-64-v2-AES (portable, compatible with most modern x86 hardware)
  Memory:
    MiB: as needed
    Ballooning: checked (allows dynamic memory adjustment with qemu-agent installed)
  Network:
    Model: VirtIO (best performance)
    Bridge: vmbr0
    VLAN Tag: set if using VLANs

VirtIO drivers

For Windows VMs, download the VirtIO ISO from the Fedora project and attach it as a second CD-ROM. During installation, load the VirtIO SCSI driver to see the disk. After installation, run the VirtIO installer from the ISO to get network and balloon drivers.

1
2
3


# Download VirtIO ISO to local storage on Proxmox
wget -P /var/lib/vz/template/iso/ \
  https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso

Cloud-init templates

Cloud-init is the standard mechanism for first-boot configuration of cloud images (hostname, SSH keys, network, users). Proxmox has native cloud-init support — it generates the cloud-init seed as a CDROM attached to the VM.

To create a reusable cloud-init template from an Ubuntu cloud image:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


# Download the cloud image
wget -P /tmp/ https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

# Create a new VM (ID 9000 by convention for templates)
qm create 9000 \
  --name ubuntu-2404-template \
  --memory 2048 \
  --cores 2 \
  --net0 virtio,bridge=vmbr0

# Import the cloud image disk to your ZFS storage
qm importdisk 9000 /tmp/noble-server-cloudimg-amd64.img vmdata

# Attach the imported disk
qm set 9000 \
  --scsihw virtio-scsi-pci \
  --scsi0 vmdata:vm-9000-disk-0,discard=on

# Add cloud-init drive
qm set 9000 --ide2 vmdata:cloudinit

# Boot from the disk
qm set 9000 --boot c --bootdisk scsi0

# Set serial console (required for cloud-init display in some images)
qm set 9000 --serial0 socket --vga serial0

# Configure cloud-init defaults
qm set 9000 \
  --ciuser ubuntu \
  --sshkeys ~/.ssh/authorized_keys \
  --ipconfig0 ip=dhcp

# Convert to template
qm template 9000

Clone the template to create new VMs instantly:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Full clone (independent copy)
qm clone 9000 101 --name webserver-01 --full --storage vmdata

# Linked clone (thin — uses template as base; shares storage with template)
qm clone 9000 102 --name webserver-02

# Resize the cloned disk
qm resize 101 scsi0 +20G

# Start
qm start 101

Linked clones start in seconds and consume only the delta from the template. They require the template disk to remain intact — deleting the template breaks all linked clones.

qemu-guest-agent

Install qemu-guest-agent in the guest for accurate IP reporting, proper shutdown/reboot from the Proxmox UI, memory ballooning, and filesystem freeze during snapshots:

1
2
3
4
5
6
7


# Debian/Ubuntu guest
apt install qemu-guest-agent
systemctl enable --now qemu-guest-agent

# RHEL/Fedora guest
dnf install qemu-guest-agent
systemctl enable --now qemu-guest-agent

Enable in the VM options:

VM → Options → QEMU Guest Agent → Enabled: checked

LXC Containers

LXC containers share the Proxmox host kernel. They start in under a second, use a fraction of the memory of a VM, and have near-native I/O performance. The trade-off: they must run Linux, they run the host’s kernel (no custom kernel modules in the container), and they have a slightly wider attack surface than a full VM (shared kernel means a container escape is more impactful than a VM escape).

When to use LXC vs KVM

Criterion	LXC	KVM
OS	Linux only	Any
Boot time	< 1 second	10–60 seconds
Memory overhead	20–50 MB	200–500 MB+
Kernel modules	Host kernel only	Guest kernel
Isolation	Namespace-based	Hardware-level
GPU passthrough	Not supported	Supported
Nested virtualization	Limited	Supported
Windows	No	Yes

Use LXC for: lightweight Linux services (DNS, proxy, monitoring agents, databases, web servers), services that do not need custom kernel modules, dev environments where boot speed matters.

Use KVM for: Windows, anything needing GPU passthrough, anything requiring a different kernel version, anything where strong isolation is a security requirement.

Creating an LXC container

Download a container template:

1
2
3
4
5


# List available templates
pveam available

# Download Ubuntu 24.04
pveam download local ubuntu-24.04-standard_24.04-2_amd64.tar.zst

Via the UI:

Create CT
  General: Node, CT ID, Hostname, Password (or SSH key)
  Template: local → ubuntu-24.04-standard
  Disks: Storage (vmdata), Size
  CPU: Cores
  Memory: RAM, Swap
  Network: Name (eth0), Bridge (vmbr0), IPv4 (DHCP or static)
  DNS: inherited from host or custom

Via CLI:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


pct create 200 local:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst \
  --hostname dns-server \
  --memory 512 \
  --swap 512 \
  --cores 1 \
  --net0 name=eth0,bridge=vmbr0,ip=192.168.1.20/24,gw=192.168.1.1 \
  --storage vmdata \
  --rootfs vmdata:8 \
  --password \
  --unprivileged 1 \
  --start 1

--unprivileged 1 runs the container with user namespace mapping — UIDs inside the container are mapped to unprivileged UIDs on the host. This is the safe default; privileged containers have root inside the container map to root on the host, which is a security risk.

Privileged vs unprivileged containers

Most containers should be unprivileged. Exceptions where privileged containers are needed:

Containers that run Docker (Docker inside LXC requires privileged or significant nesting config)
Containers that need to mount NFS inside the container
Containers using certain kernel features not available in user namespaces

For Docker inside LXC (privileged):

1

pct set 200 --features keyctl=1,nesting=1

For Docker inside LXC (unprivileged — cleaner):

1
2
3
4
5


pct set 200 --features keyctl=1,nesting=1
# Add to /etc/pve/lxc/200.conf:
# lxc.apparmor.profile: unconfined
# lxc.cgroup2.devices.allow: a
# lxc.cap.drop:

Bind mounts

Share a directory from the Proxmox host into an LXC container:

1
2
3
4
5


# In /etc/pve/lxc/200.conf
mp0: /mnt/nas/media,mp=/media,shared=1

# Or via pct
pct set 200 --mp0 /mnt/nas/media,mp=/media

This is how you give a Plex or Jellyfin container in LXC access to media stored on a host-mounted NAS share.

Snapshots and Backups

VM and container snapshots

A snapshot captures the disk state (and optionally RAM state) of a VM at a point in time. With ZFS or LVM-Thin storage, disk snapshots are nearly instant; with directory storage, they are slow and copy the full disk.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# Create a snapshot (include RAM state with --vmstate)
qm snapshot 101 before-upgrade --description "Before apt dist-upgrade"
pct snapshot 200 before-upgrade

# List snapshots
qm listsnapshot 101
pct listsnapshot 200

# Roll back
qm rollback 101 before-upgrade
pct rollback 200 before-upgrade

# Delete snapshot
qm delsnapshot 101 before-upgrade

Snapshots are not backups. A snapshot on the same disk as the VM is deleted if the storage fails. Use them for short-term safety (before risky operations) and use a backup solution for durable recovery.

Proxmox Backup Server (PBS)

PBS is a separate appliance (installed on its own machine or VM) that provides deduplicated, incremental, encrypted backups. It is the recommended backup target for Proxmox environments.

PBS works by taking a backup of each changed block since the last run — the first backup is a full backup, subsequent backups are incremental. Deduplication happens across all backups in the datastore, so similar VMs share chunks. Encryption is AES-256-GCM.

Install PBS on a dedicated machine (can be a VM on the same node or a separate box):

1
2
3
4
5
6


# On the PBS machine (Debian 12)
echo "deb http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription" \
  > /etc/apt/sources.list.d/pbs-no-subscription.list
wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg \
  -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
apt update && apt install proxmox-backup-server

PBS web UI runs on port 8007.

Add PBS as a storage backend in Proxmox:

Datacenter → Storage → Add → Proxmox Backup Server
  ID: pbs
  Server: 192.168.1.60
  Datastore: main
  Username: user@pbs
  Password: ...
  Fingerprint: (copy from PBS → Dashboard → Fingerprint)

Schedule backups:

Datacenter → Backup → Add
  Storage: pbs
  Schedule: sun 02:00 (or any systemd calendar expression)
  Selection: All (or select specific VMs/CTs)
  Mode: Snapshot (for running VMs)
  Compression: zstd
  Encryption: enabled (set encryption key)
  Retention: Keep Last 7, Keep Weekly 4, Keep Monthly 6

The retention policy uses garbage collection to remove chunks not referenced by any retained backup.

vzdump for direct backups

vzdump is Proxmox’s backup tool used by both the UI scheduler and PBS. Useful for ad-hoc backups:

1
2
3
4
5
6
7
8


# Backup VM 101 to local storage (creates .vma.zst file)
vzdump 101 --storage local --compress zstd --mode snapshot

# Backup all VMs and containers
vzdump --all --storage pbs --compress zstd --mode snapshot

# Backup with bandwidth limit (MB/s)
vzdump 101 --storage pbs --bwlimit 100

Clustering

A Proxmox cluster is a group of nodes managed as a single unit. You can create and manage VMs on any node from any node’s web UI, migrate VMs between nodes, and — with shared storage — enable high availability.

Requirements

Minimum 3 nodes for reliable quorum (2 nodes cannot handle one failure without external quorum device)
All nodes must resolve each other’s hostnames
Low-latency network for Corosync (< 2ms recommended; separate physical link preferred)
Same Proxmox version on all nodes
Same time (NTP synchronized)

Creating the cluster

On the first node:

1
2


pvecm create my-cluster --link0 192.168.10.1
# link0 is the Corosync network address for this node (use dedicated NIC/VLAN)

On each additional node:

1
2


pvecm add 192.168.10.1 --link0 192.168.10.2
# This node's Corosync address is 192.168.10.2

Verify:

1
2


pvecm status
pvecm nodes

With a cluster in place, the web UI shows all nodes in the left panel. VMs can be created on any node and migrated between them.

Corosync redundancy

Configure a second Corosync link for redundancy. Both links must be alive for the node to be healthy; losing one generates a warning but the cluster continues operating:

1
2


pvecm add 192.168.10.1 --link0 192.168.10.2 --link1 10.10.10.2
# link1 uses a separate NIC for redundancy

Quorum and split-brain

Quorum requires more than half of nodes to be reachable. In a 3-node cluster, losing 1 node leaves 2 nodes with quorum (2 > 3/2). In a 2-node cluster, losing 1 node leaves 1 node without quorum — the remaining node pauses to prevent split-brain (both nodes believing they are the primary and potentially corrupting shared data).

For 2-node clusters, add a QDevice (Quorum Device) — a lightweight arbitrator that breaks ties without being a full cluster member:

1
2
3
4
5


# On a separate machine (can be a Raspberry Pi)
apt install corosync-qnetd

# On both cluster nodes
pvecm qdevice setup <qdevice-ip>

Live migration

Migrate a running VM between nodes:

1
2
3
4
5
6
7
8


# Online migration (no downtime — VM continues running during transfer)
qm migrate 101 pve2 --online

# Offline migration (VM is stopped, moved, restarted)
qm migrate 101 pve2

# Migrate LXC container
pct migrate 200 pve2 --restart

Online migration transfers RAM state over the migration network while the VM runs. At the end, a brief pause synchronizes the last dirty pages. The pause is typically under 1 second for VMs under moderate memory write pressure.

For online migration to work without shared storage, Proxmox copies the disk from the source node to the destination over the migration network (offline copy-then-start, not true live migration). True live migration with zero downtime requires shared storage (NFS, Ceph) accessible from all nodes.

High Availability

Proxmox HA automatically restarts VMs and containers on another node if their current node fails. HA requires a cluster with shared storage.

Enabling HA for a VM

Datacenter → HA → Resources → Add
  VM: 101
  State: started
  Group: (optional — restrict to specific nodes)
  Max Restart: 3
  Max Relocate: 1

Or via CLI:

1

ha-manager add vm:101 --state started --max_restart 3

When the node hosting VM 101 becomes unavailable, pve-ha-crm (cluster resource manager) detects the failure via Corosync, fences the failed node if necessary (via IPMI/BMC), and starts the VM on another node.

Fencing (STONITH)

Without fencing, HA is unsafe. If a node loses network connectivity but is still running, it might still be writing to shared storage while HA starts the VM on another node — two instances writing to the same disk simultaneously corrupts data. Fencing ensures the failed node is powered off before HA starts the VM elsewhere.

Configure fencing via IPMI/BMC:

Datacenter → HA → Fencing → Add
  Plugin: ipmi
  Node: pve2
  Address: 192.168.1.201 (IPMI address)
  Username: admin
  Password: ...

Without working fencing in production, HA is a liability not an asset.

GPU Passthrough

PCI passthrough assigns a physical GPU (or any PCIe device) exclusively to a VM. The VM gets direct hardware access — no virtualization overhead, full performance, full driver support.

Verify IOMMU groups

After enabling IOMMU (described in the installation section), check that your GPU is in its own IOMMU group:

1
2
3
4


for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf "Group %s: %s\n" "$n" "$(lspci -nns ${d##*/})"
done | grep -i "vga\|3d\|display\|audio"

If the GPU shares an IOMMU group with other devices (common on consumer-grade motherboards), you must pass through all devices in the group, or apply the ACS override patch (which has security implications).

Bind GPU to VFIO driver

The GPU must be detached from the host driver and handed to VFIO before a VM can use it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# Find GPU PCI IDs
lspci -nn | grep -i "nvidia\|amd\|radeon"
# Example: 01:00.0 VGA [10de:2684] (NVIDIA RTX 4090)
#          01:00.1 Audio [10de:22ba]

# Bind to VFIO at boot — add to /etc/modprobe.d/vfio.conf
echo "options vfio-pci ids=10de:2684,10de:22ba" > /etc/modprobe.d/vfio.conf

# Blacklist NVIDIA/AMD driver from claiming the GPU
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf

update-initramfs -u -k all
reboot

# Verify VFIO has claimed the GPU
lspci -k -s 01:00.0 | grep "Kernel driver"
# Should show: Kernel driver in use: vfio-pci

Add GPU to VM

VM → Hardware → Add → PCI Device
  Device: 0000:01:00.0 (your GPU)
  All Functions: checked (passes GPU + audio together)
  ROM-Bar: checked
  PCI-Express: checked (for x16 bandwidth)

For NVIDIA GPUs in VMs, you must hide the KVM hypervisor flag (NVIDIA drivers refuse to load if they detect they are running in a VM):

1
2


qm set 101 --cpu host,hidden=1
qm set 101 --args '-cpu host,kvm=off'

Or in the VM config file (/etc/pve/qemu-server/101.conf):

cpu: host,hidden=1
args: -cpu host,kvm=off

Install NVIDIA drivers normally inside the VM — it sees the GPU as bare metal hardware.

Automation: REST API and Terraform

REST API

Every Proxmox operation is available via REST API. Get an API token:

User → API Tokens → Add
  User: root@pam
  Token ID: automation
  Privilege Separation: unchecked (inherits user permissions)

Use the token:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


# List VMs on a node
curl -s -k \
  -H "Authorization: PVEAPIToken=root@pam!automation=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
  https://pve1:8006/api2/json/nodes/pve1/qemu | jq '.data[] | {vmid, name, status}'

# Create a VM
curl -s -k -X POST \
  -H "Authorization: PVEAPIToken=root@pam!automation=..." \
  -H "Content-Type: application/json" \
  -d '{"vmid":200,"name":"test-vm","memory":2048,"cores":2}' \
  https://pve1:8006/api2/json/nodes/pve1/qemu

# Clone a template
curl -s -k -X POST \
  -H "Authorization: PVEAPIToken=root@pam!automation=..." \
  -d 'newid=201&name=new-vm&full=1&storage=vmdata' \
  https://pve1:8006/api2/json/nodes/pve1/qemu/9000/clone

Terraform with the Proxmox provider

The bpg/proxmox Terraform provider is the most complete and actively maintained option:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


# versions.tf
terraform {
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "~> 0.60"
    }
  }
}

provider "proxmox" {
  endpoint = "https://pve1:8006/"
  api_token = "root@pam!automation=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  insecure = false   # set true if using self-signed cert
  ssh {
    agent    = true
    username = "root"
  }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


# vm.tf — clone from cloud-init template
resource "proxmox_virtual_environment_vm" "webserver" {
  name      = "webserver-01"
  node_name = "pve1"
  vm_id     = 101

  clone {
    vm_id   = 9000   # template ID
    full    = true
    node_name = "pve1"
  }

  cpu {
    cores   = 2
    type    = "host"
  }

  memory {
    dedicated = 4096
  }

  disk {
    datastore_id = "vmdata"
    interface    = "scsi0"
    size         = 32
    discard      = "on"
    file_format  = "raw"
  }

  network_device {
    bridge  = "vmbr0"
    model   = "virtio"
    vlan_id = 20
  }

  initialization {
    ip_config {
      ipv4 {
        address = "192.168.20.101/24"
        gateway = "192.168.20.1"
      }
    }
    user_account {
      username = "ubuntu"
      keys     = [file("~/.ssh/id_ed25519.pub")]
    }
  }

  agent {
    enabled = true
  }

  operating_system {
    type = "l26"   # Linux 2.6+ kernel
  }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


# lxc.tf — create an LXC container
resource "proxmox_virtual_environment_container" "dns" {
  description = "Unbound DNS resolver"
  node_name   = "pve1"
  vm_id       = 201

  initialization {
    hostname = "dns-01"
    ip_config {
      ipv4 {
        address = "192.168.1.20/24"
        gateway = "192.168.1.1"
      }
    }
    user_account {
      password = var.container_password
    }
  }

  cpu {
    cores = 1
  }

  memory {
    dedicated = 512
    swap      = 512
  }

  disk {
    datastore_id = "vmdata"
    size         = 8
  }

  network_interface {
    name   = "eth0"
    bridge = "vmbr0"
  }

  operating_system {
    template_file_id = "local:vztmpl/ubuntu-24.04-standard_24.04-2_amd64.tar.zst"
    type             = "ubuntu"
  }

  unprivileged = true
  start_on_boot = true
}

Useful CLI Commands

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38


# VM management
qm list                          # list all VMs
qm status 101                    # VM status
qm start 101                     # start
qm shutdown 101                  # graceful shutdown (via guest agent)
qm stop 101                      # force stop
qm reset 101                     # hard reset
qm suspend 101                   # suspend to RAM
qm resume 101
qm config 101                    # show VM config
qm set 101 --memory 8192         # change memory (hot-plug if supported)
qm terminal 101 -iface serial0   # connect to serial console
qm monitor 101                   # QEMU monitor (advanced)

# Container management
pct list
pct status 200
pct start 200
pct shutdown 200
pct stop 200
pct config 200
pct set 200 --memory 1024
pct enter 200                    # open shell inside container

# Storage
pvesm status                     # storage pool status
pvesm list vmdata                # list content of a storage pool
pvesm alloc vmdata 101 disk-101  16G  # allocate a disk

# Cluster
pvecm status
pvecm nodes
ha-manager status
ha-manager add vm:101 --state started
ha-manager remove vm:101

# Backup
vzdump 101 --storage pbs --mode snapshot --compress zstd

Honest Trade-offs

What Proxmox does well. The convergence of VMs, containers, storage, networking, clustering, HA, and backup in a single platform with a coherent UI is genuinely valuable. ZFS integration is first-class. The API is complete. For a homelab or SMB environment, it is an extraordinary amount of capability for zero licensing cost. The community is large and the documentation (the official admin guide, pve.proxmox.com/pve-docs) is thorough.

Subscription model. Proxmox is free to use but the stable enterprise repository and official support require a subscription (€105–€1,050/node/year depending on tier). The no-subscription repository works fine for homelab use and most small deployments. For production environments where upstream response time matters, a subscription is worth considering — it is cheap compared to VMware licensing.

LXC security caveats. Unprivileged LXC containers are reasonably secure but the shared kernel is a broader attack surface than a KVM VM. Container escapes, while uncommon, are not impossible and have a higher impact than VM escapes. For workloads handling sensitive data or exposed to external input, KVM VMs are the conservative choice.

Ceph operational complexity. Ceph is powerful and scales well, but it is not simple to operate. A degraded Ceph cluster can leave VMs inaccessible until the cluster heals. For a homelab with 2–3 nodes, ZFS on NFS is simpler and recoverable without understanding CRUSH maps and PG states. Ceph makes sense when you have 3+ nodes each with multiple dedicated storage disks and someone who understands how to recover it.

GPU passthrough caveats. The GPU is exclusively assigned to one VM. If the VM is off, the GPU is idle and unavailable to the host or other VMs. NVIDIA driver version matching between host (for VFIO) and guest (for computation) sometimes requires careful selection. Consumer NVIDIA GPUs (RTX series) do not support NVIDIA vGPU (which would allow sharing across VMs) without workarounds. If you need multiple VMs with GPU access, you need multiple GPUs or an enterprise card with SR-IOV support.

Migration network bottleneck. Without shared storage (Ceph or NFS), live migration copies the full disk over the migration network. A 100 GB VM disk on a 1GbE link takes over 10 minutes to copy — during which the VM is running on the source but will pause at the end for final synchronization. Plan your migration network for this.