Proxmox + Ceph: Hyperconverged Storage Deep Dive

Hyperconverged infrastructure (HCI) collapses separate compute and storage servers into a single cluster where every node contributes both CPU/RAM for VMs and disk capacity to a shared storage pool. Proxmox VE with Ceph is the leading open-source HCI stack — and unlike commercial alternatives (Nutanix, VMware vSAN), it runs on commodity hardware with no per-socket licensing.

This guide goes deep: how Ceph works internally, how to make correct design decisions before the first disk is formatted, how to tune for SSDs versus spinning disks, and how to operate the cluster reliably in production.

Why Ceph, and Why With Proxmox

Ceph is a distributed storage system that provides object storage (RADOS), block storage (RBD), and filesystem storage (CephFS) from the same pool of disks. For Proxmox VM storage, RBD (RADOS Block Device) is the relevant interface — it exposes virtual block devices backed by the Ceph cluster.

What makes Ceph valuable for Proxmox:

• No single point of failure: data is replicated across multiple nodes; any node can die and VMs keep running
• No separate storage network required: Ceph can run on the same nodes as Proxmox, though a dedicated network is strongly recommended
• Live migration without shared SAN: since all nodes access the same RBD pool, Proxmox can live-migrate VMs between nodes without a separate SAN appliance
• Scales linearly: add nodes and OSDs to increase capacity and throughput

What Ceph is not:

• Not a low-latency storage system: Ceph’s distributed architecture introduces latency compared to local NVMe. Expect 1–3ms for writes vs <0.1ms for local NVMe. For latency-sensitive databases, local storage with VM replication may be better.
• Not simple to operate: Ceph has many failure modes and requires understanding its architecture to debug correctly

Architecture Overview

Core Components

MON (Monitor): maintains the cluster map — the authoritative record of what’s in the cluster, where everything is, and the current health state. Monitors use Paxos consensus; you need an odd number (3, 5) for quorum. On a 3-node Proxmox cluster, each node runs one MON.

MGR (Manager): handles metrics, the dashboard, and orchestration modules. Runs alongside MONs, typically one per node.

OSD (Object Storage Daemon): one OSD process per physical disk. Each OSD stores data objects, replicates to peer OSDs, and participates in recovery. The OSD is the fundamental storage unit in Ceph.

MDS (Metadata Server): only needed for CephFS (filesystem interface). Not required for RBD (block storage) used by Proxmox VMs.

How Data Gets Stored: CRUSH

When a client writes data, Ceph doesn’t have a central metadata server that tracks where each piece lives. Instead, it uses a deterministic algorithm called CRUSH (Controlled Replication Under Scalable Hashing) to compute placement:

object_name → hash → placement group (PG) → CRUSH map → OSDs

Given any object name, every node in the cluster independently computes the same answer for which OSDs hold that object. No lookups required, no central coordinator bottleneck.

Placement Groups (PGs): objects are first mapped to PGs (a logical bucket), then PGs are mapped to OSDs. This two-level indirection means rebalancing only requires moving PGs between OSDs, not individual objects.

The CRUSH map describes your hardware topology: datacenters → rooms → racks → hosts → OSDs. CRUSH uses this topology to ensure replicas are placed on failure-independent hardware — different hosts, different racks, potentially different datacenters.

Hardware Design

Minimum vs Recommended Configurations

Absolute minimum (lab/homelab):

• 3 nodes (for MON quorum and replica-3)
• 1 OSD per node (3 OSDs total)
• All-in-one: MON + MGR + OSD on each node
• Works but has no fault tolerance beyond single-node failure

Small production (5–20 VMs):

• 3 nodes
• 2–4 OSDs per node (6–12 OSDs total)
• Dedicated Ceph cluster network (10 GbE minimum)
• SSD WAL/DB devices, HDD data devices

Medium production (20–100 VMs):

• 4–6 nodes (allows rolling maintenance without degraded state)
• 4–8 SSDs per node
• 25 GbE cluster network
• Separate network for Ceph public and cluster traffic

The Three Networks

Ceph uses (or should use) three separate networks:

Separating public from cluster traffic is critical for performance. When a drive fails and Ceph starts recovering data, cluster traffic can saturate a 10 GbE link. Without a separate cluster network, VM I/O competes with recovery traffic.

1
2
3
4

# /etc/ceph/ceph.conf
[global]
public_network = 10.0.10.0/24    # VM I/O and management
cluster_network = 10.0.20.0/24   # OSD replication (separate physical NICs)

OSD Device Selection

All-NVMe: highest performance, simplest configuration. Each NVMe drives its own OSD. For VM workloads, this is the best option if the budget allows.

NVMe WAL/DB + HDD data (hybrid): HDDs for capacity, NVMe for the write-ahead log and RocksDB metadata. Dramatically improves HDD performance. Each NVMe can back 4–6 HDD OSDs.

All-HDD: lowest cost per TB. Acceptable for bulk storage, backup targets, or archival. Not suitable for latency-sensitive VM workloads.

All-SSD (SATA): good middle ground. SATA SSDs at 2–4 TB are cost-effective and deliver consistent IOPS without the complexity of hybrid configurations.

Deploying Ceph with Proxmox

Proxmox includes a Ceph installation wizard in the web UI. For production deployments, understanding the underlying commands matters more than the wizard.

Initial Setup

On each node, install Ceph:

1
2
3

# In the Proxmox web UI: Node → Ceph → Install Ceph
# Or via CLI on each node:
pveceph install --repository no-subscription

Initialize the first monitor:

1
2
3
4
5

# On node 1
pveceph init --network 10.0.10.0/24

# Optionally specify cluster network separately
pveceph init --network 10.0.10.0/24 --cluster-network 10.0.20.0/24

Add monitors on the other nodes (via UI: each node → Ceph → Monitor → Create):

1
2
3

# On node 2 and node 3
pveceph createmon
pveceph createmgr

Creating OSDs

Using the Proxmox web UI: Node → Ceph → OSD → Create OSD. Select the disk, optionally set WAL/DB devices.

Via CLI:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Simple OSD on a single NVMe
pveceph createosd /dev/nvme0n1

# OSD with dedicated WAL and DB on a faster NVMe
pveceph createosd /dev/sdb \
  --wal-dev /dev/nvme0n1 \
  --db-dev /dev/nvme0n1

# List OSD tree
ceph osd tree

OSD BlueStore Configuration

BlueStore (the default Ceph storage backend since Ceph Luminous) uses RocksDB for its metadata store. The WAL and DB can live on the same device as data or be separated onto faster media.

Device layout options:

1. All on HDD (slowest):
   [  HDD: data | DB (2-3% of disk) | WAL (512MB-1GB) ]

2. DB on SSD, WAL on SSD (recommended for HDD OSDs):
   [  HDD: data  ] + [  SSD: DB + WAL  ]

3. All on NVMe (fastest, simplest):
   [  NVMe: data + DB + WAL  ]

For hybrid configurations, size the DB partition:

1
2
3

# DB size rule of thumb: 2% of OSD capacity (min 1 GB, max 30 GB)
# For a 4 TB HDD OSD: 4000 GB × 2% = 80 GB DB partition
# For a 10 TB HDD OSD: 10000 GB × 2% = 200 GB (cap at 30 GB if constrained)

CRUSH Maps

The CRUSH map is where most Ceph tuning happens. Proxmox creates a basic CRUSH map automatically, but production clusters need customization.

Viewing the Current CRUSH Map

1
2
3
4
5
6
7

# Get the compiled CRUSH map
ceph osd getcrushmap -o crushmap.bin

# Decompile to human-readable format
crushtool -d crushmap.bin -o crushmap.txt

cat crushmap.txt

Output structure:

# devices
device 0 osd.0 class ssd
device 1 osd.1 class ssd
device 2 osd.2 class ssd
device 3 osd.3 class hdd
...

# types
type 0 osd
type 1 host
type 2 rack
type 3 datacenter
type 4 root

# buckets
host pve1 {
    id -2
    alg straw2
    item osd.0 weight 0.875    # 1 TB NVMe = 0.875 TiB
    item osd.1 weight 0.875
}
host pve2 {
    id -3
    alg straw2
    item osd.2 weight 0.875
    item osd.3 weight 0.875
}
host pve3 {
    id -4
    alg straw2
    item osd.4 weight 0.875
    item osd.5 weight 0.875
}

root default {
    id -1
    alg straw2
    item pve1 weight 1.750
    item pve2 weight 1.750
    item pve3 weight 1.750
}

# rules
rule replicated_rule {
    id 0
    type replicated
    step take default
    step chooseleaf firstn 0 type host  # place replicas on different hosts
    step emit
}

Adding Rack Awareness

For clusters spanning multiple racks, add rack-level CRUSH buckets to ensure replicas land on different racks:

# Edit crushmap.txt

# Add rack buckets
rack rack1 {
    id -10
    alg straw2
    item pve1 weight 1.750
    item pve2 weight 1.750
}
rack rack2 {
    id -11
    alg straw2
    item pve3 weight 1.750
    item pve4 weight 1.750
}

# Update root
root default {
    id -1
    alg straw2
    item rack1 weight 3.500
    item rack2 weight 3.500
}

# Update rule to fail at rack level
rule replicated_rule {
    id 0
    type replicated
    step take default
    step chooseleaf firstn 0 type rack  # replicas on different racks
    step emit
}

1
2
3

# Recompile and inject
crushtool -c crushmap.txt -o crushmap-new.bin
ceph osd setcrushmap -i crushmap-new.bin

Device Classes

Ceph 12+ supports device classes (ssd, hdd, nvme) that allow you to create pools pinned to specific hardware tiers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# List device classes
ceph osd tree | grep class

# Set/override a device class
ceph osd crush set-device-class nvme osd.0 osd.1 osd.2
ceph osd crush set-device-class hdd osd.3 osd.4 osd.5

# Create a CRUSH rule for a specific class
ceph osd crush rule create-replicated ssd-rule default host ssd
ceph osd crush rule create-replicated hdd-rule default host hdd

# Verify
ceph osd crush rule dump ssd-rule

Pools and Replication

Replicated Pools

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Create a pool for VM images (RBD)
ceph osd pool create vm-pool replicated ssd-rule

# Set replication factor (default is 3 — keep this for production)
ceph osd pool set vm-pool size 3          # total copies
ceph osd pool set vm-pool min_size 2      # minimum copies for writes to succeed

# Enable RBD application on the pool
ceph osd pool application enable vm-pool rbd

# Initialize RBD
rbd pool init vm-pool

Erasure Coding

Erasure coding stores data with parity rather than full replicas. An EC 4+2 pool stores 4 data chunks and 2 parity chunks — any 2 of the 6 chunks can be lost and the data is recoverable.

Storage efficiency comparison:

• Replica-3: 1 TB of data → 3 TB raw storage (33% efficiency)
• EC 4+2: 1 TB of data → 1.5 TB raw storage (67% efficiency)
• EC 8+3: 1 TB of data → 1.375 TB raw storage (73% efficiency)

Tradeoffs:

• EC requires more OSDs to distribute chunks (4+2 needs at least 6 OSDs on 6 different nodes)
• EC reads are slower for partial reads (must decode from chunks)
• EC writes require read-modify-write for partial writes — poor for random write workloads
• EC is best for bulk storage: backup targets, cold data, object storage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# Create an EC profile
ceph osd erasure-code-profile set ec-4-2 \
  k=4 m=2 \
  crush-failure-domain=host \
  plugin=jerasure \
  technique=reed_sol_van

# Create EC pool
ceph osd pool create bulk-pool erasure ec-4-2

# EC pools can't be used directly for RBD (no partial writes)
# Create a replicated overlay cache pool for the metadata
ceph osd pool create bulk-pool-meta replicated ssd-rule
ceph osd pool set bulk-pool-meta size 3

# For CephFS or object storage — EC works directly
ceph fs add_data_pool cephfs bulk-pool

For VM storage: use replicated pools. For backup storage or bulk data: EC is appropriate.

Placement Group (PG) Sizing

Incorrectly sized PG counts cause uneven data distribution or excessive memory use on OSDs. The formula:

Target PGs per OSD = 100 (for < 5 pools) to 200 (for many pools)
PG count = (OSDs × Target PGs per OSD) / pool_size
Round up to nearest power of 2

Example: 12 OSDs, 2 pools, replica-3:

PGs = (12 × 100) / 3 = 400 → round to 512
Each pool gets 256 PGs (half of total)

1
2
3
4
5
6
7
8
9

# Set PG count when creating pool
ceph osd pool create vm-pool 256

# Enable pg_autoscaler (Ceph Nautilus+) to manage PG counts automatically
ceph mgr module enable pg_autoscaler
ceph osd pool set vm-pool pg_autoscale_mode on

# Check autoscaler recommendations
ceph osd pool autoscale-status

The autoscaler is generally safe to enable in production — it adjusts PG counts gradually without disruption.

Adding Proxmox Storage to Ceph

Once the pool exists, add it to Proxmox as a storage target:

1
2
3
4
5
6
7
8

# Via CLI
pvesm add rbd ceph-vms \
  --monhost "10.0.10.1,10.0.10.2,10.0.10.3" \
  --pool vm-pool \
  --username admin \
  --content images,rootdir

# Or via Web UI: Datacenter → Storage → Add → RBD

Now VMs can be created with disks on the Ceph pool, and live migration between any nodes in the cluster works automatically.

Tuning for Performance

SSD/NVMe Tuning

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

# /etc/ceph/ceph.conf

[osd]
# Disable rotational flag for SSDs
osd_journal_size = 0                    # not used with BlueStore

# BlueStore cache — increase for NVMe
bluestore_cache_size_ssd = 4294967296   # 4 GB per OSD (if RAM allows)
bluestore_cache_size_hdd = 1073741824   # 1 GB per OSD for HDD

# Queue depth
bluestore_rocksdb_options = max_write_buffer_number=4,\
  min_write_buffer_number_to_merge=1,\
  recycle_log_file_num=4,\
  write_buffer_size=268435456,\
  writable_file_max_buffer_size=0,\
  compaction_readahead_size=2097152

# OSD scrubbing schedule (avoid peak hours)
osd_scrub_begin_hour = 2
osd_scrub_end_hour = 6
osd_deep_scrub_interval = 604800        # weekly deep scrub

Network Tuning

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

[global]
# Increase network buffer sizes for high-throughput clusters
ms_dispatch_throttle_bytes = 1073741824  # 1 GB
osd_client_message_size_cap = 524288000  # 500 MB

# Enable jumbo frames if your switch supports it
# (configure MTU 9000 on all cluster interfaces first)
ms_bind_msgr1 = true

[osd]
osd_recovery_max_active_hdd = 3
osd_recovery_max_active_ssd = 10
osd_max_backfills = 2                   # concurrent backfills per OSD
osd_recovery_sleep_hdd = 0.1           # throttle recovery to protect VM I/O
osd_recovery_sleep_ssd = 0

RBD Client Tuning

On the Proxmox node side, tune the RBD client parameters for VM workloads:

1
2
3
4
5
6
7
8
9

# /etc/ceph/ceph.conf

[client]
rbd_cache = true
rbd_cache_size = 134217728              # 128 MB RBD client cache
rbd_cache_max_dirty = 100663296         # 96 MB max dirty
rbd_cache_target_dirty = 67108864       # 64 MB target dirty
rbd_cache_max_dirty_age = 1.0
rbd_cache_writethrough_until_flush = true

QoS for VM Pools

Prevent a single VM from saturating the cluster:

1
2
3
4
5
6

# Set IOPS limits on an RBD image
rbd config image set vm-pool/vm-100-disk-0 rbd_qos_iops_limit 5000
rbd config image set vm-pool/vm-100-disk-0 rbd_qos_bps_limit 524288000  # 500 MB/s

# Set pool-wide defaults
ceph config set client rbd_qos_iops_limit 10000

Monitoring and Alerting

Cluster Health

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# Overall health
ceph status
ceph health detail

# OSD status
ceph osd stat
ceph osd perf             # per-OSD latency and throughput

# Pool statistics
ceph df                   # usage by pool
rados df                  # object count and size

# PG status
ceph pg stat
ceph pg dump | grep -v "^pg_stat" | awk '{print $1, $15}' | sort -k2  # PGs by state

Prometheus Metrics

The Ceph MGR ships a Prometheus module:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Enable Prometheus metrics endpoint
ceph mgr module enable prometheus

# Endpoint is now available at:
# http://<mgr-host>:9283/metrics

# Key metrics to alert on:
# ceph_health_status > 0                      # cluster not healthy
# ceph_osd_up == 0                            # OSD down
# ceph_osd_in == 0                            # OSD out of cluster
# ceph_pg_degraded > 0                        # degraded PGs (missing replicas)
# ceph_pg_undersized > 0                      # undersized PGs
# increase(ceph_osd_recovery_ops[5m]) > 0    # active recovery

Sample Prometheus alert rules:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

groups:
  - name: ceph
    rules:
      - alert: CephHealthCritical
        expr: ceph_health_status == 2
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Ceph cluster is in HEALTH_ERR state"

      - alert: CephOSDDown
        expr: ceph_osd_up == 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Ceph OSD {{ $labels.ceph_daemon }} is down"

      - alert: CephPoolNearFull
        expr: ceph_pool_percent_used > 75
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Ceph pool {{ $labels.name }} is {{ $value }}% full"

      - alert: CephPoolFull
        expr: ceph_pool_percent_used > 85
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Ceph pool {{ $labels.name }} is critically full — writes will stop at 95%"

Common Operations

Adding a New OSD (Expanding Capacity)

1
2
3
4
5
6

# Add new disk to existing node
pveceph createosd /dev/nvme1n1

# Ceph automatically begins rebalancing
ceph status  # watch "misplaced" objects decrease
ceph osd df  # verify balanced distribution

Removing a Failing OSD

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# Mark OSD out (triggers rebalancing to other OSDs)
ceph osd out osd.5

# Wait for rebalancing to complete
watch ceph status

# Once rebalanced, stop and remove
systemctl stop ceph-osd@5
ceph osd purge osd.5 --yes-i-really-mean-it
ceph osd crush remove osd.5

# Wipe the disk for reuse
wipefs -a /dev/sdX

Handling a Full Cluster

Ceph stops accepting writes at 95% full (configurable). Prevention:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Check fill level
ceph df

# Temporarily increase full ratio to give time to add capacity
ceph osd set-full-ratio 0.97
ceph osd set-backfillfull-ratio 0.92

# Delete unnecessary snapshots (common cause of surprise fullness)
rbd snap ls vm-pool/vm-100-disk-0
rbd snap rm vm-pool/vm-100-disk-0@snap-name

# Immediately after: add capacity or migrate data

Cluster Maintenance: Taking a Node Down

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Before taking a node down for maintenance
# Set noout to prevent Ceph from marking OSDs out during brief outages
ceph osd set noout

# Perform maintenance (reboot, hardware swap)
# ...

# After node is back
ceph osd unset noout

# Verify cluster recovers to HEALTH_OK
watch ceph status

Sizing Reference

Usable Capacity Calculation

Usable capacity = Raw capacity / replication_factor × efficiency_factor

For 12 × 4 TB SSDs (48 TB raw) with replica-3:
Usable = 48 TB / 3 × 0.85 = 13.6 TB

(0.85 efficiency factor accounts for BlueStore overhead and keeping 15% headroom
to avoid performance degradation above 80% full)

Performance Expectations

These are cluster-wide figures. Per-VM performance depends on PG distribution and concurrency.

RAM Requirements

Each OSD requires approximately 4–5 GB of RAM:

• 3 nodes × 4 OSDs × 5 GB = 60 GB RAM just for Ceph OSDs
• Add 2–4 GB per MON/MGR
• Add memory for VMs

Plan for 8–16 GB RAM per OSD when sizing nodes for a combined Proxmox+Ceph deployment.

Should You Use Ceph?

Ceph adds significant operational complexity. Before committing, honestly assess:

Good fit:

• 3+ nodes where you want shared storage without a separate SAN
• Budget to do it right (dedicated cluster NICs, enough RAM per OSD)
• Team with time to learn Ceph operations
• Workloads that tolerate 1–3ms storage latency

Not a good fit:

• 2-node clusters (no fault tolerance with replica-3)
• Latency-sensitive databases that need <0.5ms storage
• Small teams with limited operations bandwidth
• Budget constraints that mean cutting the cluster network

Alternatives to consider:

• NFS from a NAS: simpler, less overhead, but single point of failure
• Local ZFS + VM replication: no shared storage, but lower latency and less complexity
• iSCSI from a dedicated storage node: shared storage without Ceph’s complexity

When Ceph is the right choice, it’s genuinely excellent — the ability to grow the cluster by adding nodes, lose a node without downtime, and manage everything through the Proxmox web UI makes it a compelling platform for organizations willing to invest in understanding it.