Standalone Ceph Without Proxmox: cephadm, Pool Design, and Day-2 Operations

April 19, 2026 15 min read Data & Databases

cephstorageobject-storagedistributed-systemslinux

Ceph is the distributed storage system that quietly runs under most large-scale open source storage stacks — OpenStack clouds, CERN’s physics workloads, DigitalOcean’s Spaces, Bloomberg’s internal platforms. It’s also the storage layer most homelab users first encounter through Proxmox, which hides almost all of Ceph’s complexity behind a web UI.

This post is about what happens when you want Ceph without Proxmox: running it on plain Linux with cephadm, designing pools and CRUSH rules for your hardware, exposing RBD/RGW/CephFS, and operating it day 2. Ceph is not a casual install — you will spend a weekend learning it even in the simplest setup — but once it’s running, it’s one of the most flexible storage systems available.

What Ceph is and isn’t

Ceph is a software-defined storage cluster that takes a bunch of disks across a bunch of machines and presents three different storage APIs on top:

RBD (RADOS Block Device) — block storage, mostly used for VM disks.
RGW (RADOS Gateway) — S3 and Swift-compatible object storage.
CephFS — POSIX distributed filesystem.

All three sit on top of RADOS (Reliable Autonomic Distributed Object Store), which is the underlying object store. Everything in Ceph is RADOS objects; the three “frontends” just translate their respective APIs to RADOS calls.

The core daemons:

MON (Monitor) — maintains cluster membership and the cluster maps. Odd number, minimum 3 for production. Paxos-based consensus.
MGR (Manager) — collects metrics, runs modules (dashboard, Prometheus exporter, orchestrator). One active, one or more standbys.
OSD (Object Storage Daemon) — one per disk. Handles all actual I/O. This is where almost all cluster CPU/RAM goes.
MDS (Metadata Server) — CephFS metadata only. Not needed for RBD or RGW.
RGW — S3/Swift gateway. Optional.

Ceph is not:

A NAS in a box. Expect operational effort.
Fast on a 3-node cluster of spinning disks. Performance comes from many disks across many nodes.
A good fit for fewer than 3 nodes. 3 is the minimum for meaningful redundancy; 5 is the minimum to survive losing a node without becoming read-only.

Hardware

Ceph is unforgiving of undersized hardware and has very specific requirements:

CPU: roughly 1 core per OSD for HDD, 2 cores per OSD for NVMe. An all-flash 10-OSD node needs 20+ cores to not bottleneck.
RAM: 4 GB per OSD minimum, 8 GB recommended. 10 OSDs on a node = 80 GB just for OSDs, plus MON/MGR overhead if they live there.
Network: 10 GbE minimum. 25 GbE preferred. Two separate networks are traditional: public_network for client traffic, cluster_network for replication. Modern guidance tolerates a single high-bandwidth network.
Disks: separate OSDs from the OS disk. Don’t put OSDs on the same drive as your boot/root.
Power: redundant PSUs matter. Ceph survives node failures but rebalance traffic on unexpected loss is painful.

A practical minimum cluster:

3 nodes, each with: 4+ cores, 32 GB RAM, 1× boot SSD, 4× data drives (SSD or HDD), 10 GbE.
Expected capacity with 3× replication: 1/3 of raw capacity.
Will survive: 1 node failure comfortably, 2 node failures as a degraded state.

cephadm: the modern way

cephadm is the current official installer. It deploys containerized Ceph daemons via Podman or Docker, reads and writes the cluster state through the Monitor, and avoids the complexity of the older ceph-deploy and Ansible playbooks. Ceph 16 (Pacific) and later are cephadm-first; anything earlier is unsupported.

Bootstrap

On your first node:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Install cephadm from the Ceph repo or distro package
dnf install -y cephadm
# or: curl --silent --remote-name --location https://github.com/ceph/ceph/raw/main/src/cephadm/cephadm
# chmod +x cephadm && ./cephadm add-repo --release reef && ./cephadm install

cephadm bootstrap \
    --mon-ip 10.0.0.11 \
    --cluster-network 10.1.0.0/24 \
    --initial-dashboard-user admin \
    --initial-dashboard-password 'changeme-please'

This creates a single-node cluster with MON, MGR, and the dashboard running. Check with:

1

cephadm shell -- ceph -s

cephadm shell drops you into a container with all Ceph tools available. The ceph CLI won’t work outside the shell unless you install ceph-common on the host.

Adding nodes

From the bootstrap node, add the SSH key to the other hosts:

1
2
3
4
5


ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node02
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node03

cephadm shell -- ceph orch host add ceph-node02 10.0.0.12
cephadm shell -- ceph orch host add ceph-node03 10.0.0.13

cephadm handles container deployment on the added hosts automatically. MON daemons will be spread across the cluster by the orchestrator. Check placement:

1
2


ceph orch host ls
ceph orch ps

Deploying OSDs

Once nodes are joined, point cephadm at their unused disks:

1
2
3
4
5
6
7
8


# Show available disks on all hosts
ceph orch device ls

# Option 1: consume all available
ceph orch apply osd --all-available-devices

# Option 2: specific disks per host
ceph orch daemon add osd ceph-node02:/dev/sdb,/dev/sdc,/dev/sdd

Ceph creates BlueStore OSDs by default. BlueStore is the current OSD backend — an LVM-wrapped raw block device with an embedded RocksDB for metadata. Do not put a filesystem on OSD disks; BlueStore takes the raw device.

For mixed HDD/SSD setups, you can put the RocksDB metadata on SSD and the data on HDD:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# osd-spec.yaml
service_type: osd
service_id: hdd_with_ssd_db
placement:
  host_pattern: 'ceph-node*'
data_devices:
  rotational: 1           # pick HDDs
db_devices:
  rotational: 0           # pick SSDs for RocksDB
  size: '50GB:'           # at least 50 GB

1

ceph orch apply -i osd-spec.yaml

Sizing the DB device: BlueStore RocksDB benefits from 4% of data size for OLTP-like patterns, 1–2% for object/backup workloads. Go higher than you think you need.

Pool design

A pool is the logical container for RADOS objects. Every RBD image, every CephFS filesystem, every RGW bucket ultimately stores its data in one or more pools. Pool configuration controls two fundamental choices:

Data durability: replicated (3 copies) or erasure-coded (k+m chunks).
Placement policy: how data is spread across the cluster (CRUSH rule).

Replicated pools

Default and recommended for small clusters and for performance-sensitive workloads like RBD:

1
2
3


ceph osd pool create data 128 128 replicated
ceph osd pool set data size 3
ceph osd pool set data min_size 2

size: number of replicas. 3 is standard.
min_size: minimum number of replicas present for I/O to continue. With size=3, min_size=2, the pool stays writable while 2 replicas are up; goes read-only if only 1 is up (to avoid split-brain).

Never run size=2. The ability to distinguish “drive actually failed” from “transient network issue” requires quorum, which requires 3. Every horror story about Ceph losing data eventually traces back to size=2.

The 128 128 numbers are placement group (PG) counts. More on this below.

Erasure-coded pools

For capacity-efficient storage of large objects (backups, archival, RGW for cold data), use erasure coding:

1
2
3
4
5
6
7
8


# Create an EC profile: 4 data chunks + 2 parity chunks
ceph osd erasure-code-profile set rgw-ec-42 \
    k=4 m=2 \
    plugin=jerasure technique=reed_sol_van \
    crush-failure-domain=host

ceph osd pool create rgw-data 128 128 erasure rgw-ec-42
ceph osd pool set rgw-data allow_ec_overwrites true   # needed for RBD on EC

k=4 m=2: 4 data chunks + 2 parity chunks. Survives any 2 chunk failures. Usable capacity = 4/6 = 67%.
crush-failure-domain=host: each chunk lands on a different host. Survives full-host failures.
Common profiles: k=2,m=1 (3 nodes, 67% efficiency, 1-node failure), k=4,m=2 (6+ nodes, 67% efficiency, 2-node failure), k=8,m=3 (11+ nodes, 73% efficiency).

EC pools have write-amplification: every small write reads k chunks, computes new parity, writes k+m chunks. This makes EC pools bad for RBD VM workloads and good for RGW/CephFS bulk storage. Don’t put EC under databases.

Placement groups

A PG (placement group) is a sub-unit of a pool. When RADOS places an object, it hashes the object name to a PG, and the PG’s membership determines which OSDs hold the object. This indirection is why Ceph scales — OSD failures only require rebalancing PGs, not re-hashing every object.

PG count matters:

Too few PGs → unbalanced data distribution, some OSDs full while others empty.
Too many PGs → excessive memory use, CPU overhead, peering storms on OSD changes.

Rough target: 100 PGs per OSD across all pools. For a cluster with 30 OSDs, budget about 3000 PGs total, distributed across pools proportional to expected capacity.

Modern Ceph has PG autoscaling enabled by default — you no longer need to calculate PG counts manually for new pools:

1
2


ceph osd pool set <pool> pg_autoscale_mode on
ceph osd pool autoscale-status

Leave autoscale on for new clusters. For clusters migrated from manual PG management, test before switching or you’ll trigger large rebalances.

CRUSH rules: where data lives

CRUSH is Ceph’s placement algorithm. Instead of storing a mapping of object → OSD (which would be enormous), CRUSH is a deterministic function: given an object hash and the current cluster topology, compute which OSDs hold the replicas. This lets any client figure out placement without querying a central metadata server.

The CRUSH map describes your cluster topology: hosts, racks, rows, datacenters. CRUSH rules say things like “place 3 replicas, each on a different host, each in a different rack”. For a typical 3-node homelab cluster, the default replicated_rule (replicate across hosts) is correct. For larger deployments, you craft rules that match your failure domains.

View and edit:

1
2
3
4
5


ceph osd getcrushmap -o crush.bin
crushtool -d crush.bin -o crush.txt
# edit crush.txt
crushtool -c crush.txt -o new-crush.bin
ceph osd setcrushmap -i new-crush.bin

The text format has buckets (hosts, racks) and rules. For most deployments, manipulate it with ceph osd crush commands rather than editing the binary:

1
2
3


ceph osd crush add-bucket rack1 rack
ceph osd crush move rack1 root=default
ceph osd crush move ceph-node02 rack=rack1

Device classes

Ceph auto-tags OSDs as hdd, ssd, or nvme. Use this for mixed-device clusters:

1
2
3
4
5
6
7


# Replicated pool that only places on SSDs
ceph osd crush rule create-replicated fast_rule default host ssd
ceph osd pool create rbd-fast 128 128 replicated fast_rule

# Similar for HDDs
ceph osd crush rule create-replicated bulk_rule default host hdd
ceph osd pool create cephfs-bulk 128 128 replicated bulk_rule

This gives you the equivalent of storage classes: fast pools for VM disks, bulk pools for archival, coexisting on the same cluster.

Frontends

RBD (block device)

Most common use case: block devices for VMs or Kubernetes PVCs.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


ceph osd pool create rbd 128 128 replicated fast_rule
ceph osd pool application enable rbd rbd
rbd pool init rbd

# Create a 50 GB image
rbd create --size 50G --pool rbd myvm-disk1

# Show info
rbd info rbd/myvm-disk1

# On the client (with the kernel rbd module)
rbd map rbd/myvm-disk1
mkfs.xfs /dev/rbd0
mount /dev/rbd0 /mnt/myvm-disk1

For Kubernetes, install the rook operator or the ceph-csi driver and point it at your cluster credentials. RBD under Kubernetes is one of the most battle-tested Ceph patterns.

Features to know:

--image-feature layering — enables COW clones (default).
--image-feature fast-diff — speeds up incremental backups.
--image-feature object-map — tracks allocated objects, speeds up many operations but breaks down for huge sparse images.

For mirroring RBD images between clusters (DR), set up rbd-mirror daemon and mirroring mode per-pool or per-image. Production-ready, asynchronous.

RGW (S3/Swift)

1

ceph orch apply rgw myorg --placement="count:2"

Deploys two RGW daemons distributed across the cluster. They act as stateless S3 endpoints — put HAProxy or any TCP load balancer in front for HA.

Create a user and bucket:

1
2
3
4
5
6


radosgw-admin user create --uid=alice --display-name="Alice" --email=alice@example.com
# Output includes access_key and secret_key

# Now use any S3 client:
aws --endpoint-url http://rgw.example.com:80 s3 mb s3://mybucket
aws --endpoint-url http://rgw.example.com:80 s3 cp file.txt s3://mybucket/

RGW has its own pool layout — it creates several pools for buckets, bucket indexes, usage logs, data. Inspect with ceph df after creating the first user:

POOL                       USED
default.rgw.buckets.data   (your actual object data)
default.rgw.buckets.index  (bucket → object listings)
default.rgw.log            (usage tracking)
default.rgw.meta           (user and bucket metadata)

Size the buckets.data pool to the bulk of your data and consider putting it on an EC pool. The index and meta pools should stay on fast SSDs (replicated) — index operations are latency-sensitive.

CephFS (POSIX)

The trickiest frontend, because distributed POSIX metadata is hard.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Deploy an MDS
ceph orch apply mds myfs --placement="count:2"

# Create data and metadata pools
ceph osd pool create cephfs_data 64 64 replicated
ceph osd pool create cephfs_metadata 32 32 replicated
ceph fs new myfs cephfs_metadata cephfs_data

# Client side — kernel mount
mount -t ceph cephmon01,cephmon02,cephmon03:/ /mnt/cephfs \
    -o name=admin,secretfile=/etc/ceph/admin.secret

CephFS scales metadata horizontally via multiple active MDS daemons (multi-MDS). For a single home server, one MDS is fine; for large clusters with many clients, scale MDS count and use subtree pinning to avoid contention:

1
2


setfattr -n ceph.dir.pin -v 0 /mnt/cephfs/team-a
setfattr -n ceph.dir.pin -v 1 /mnt/cephfs/team-b

This pins specific directory subtrees to specific MDS ranks, avoiding cross-MDS operations for those subtrees.

Snapshots and quotas on CephFS are real, work well, and are first-class — they’re among the reasons people choose CephFS over NFS for large shared filesystems.

Day-2 operations

Health checks

1
2
3
4
5


ceph -s          # brief cluster status
ceph health detail   # verbose
ceph osd df tree     # per-OSD usage
ceph osd perf        # per-OSD latency
ceph pg dump_stuck   # stuck PGs

The most important signal: ceph -s should say HEALTH_OK. Any warnings deserve investigation. Common warnings:

OSD_DOWN: an OSD has failed.
PG_DEGRADED: some PGs don’t have full replica count. Happens during rebalance; healthy if temporary.
SLOW_OPS: operations taking longer than threshold. Usually a slow or failing disk.
POOL_NEAR_FULL: 85% full by default; alert threshold.
POOL_FULL: 95% full; I/O will pause.

Replacing a failed OSD

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Mark the OSD out to trigger rebalance
ceph osd out osd.5

# Wait for rebalance to complete
watch ceph -s

# Purge the OSD
ceph osd purge 5 --yes-i-really-mean-it

# Physically replace the drive, then redeploy
ceph orch daemon add osd ceph-node02:/dev/sdX

Ceph will backfill the new OSD automatically. For large clusters, ceph config set osd osd_max_backfills 2 throttles backfill traffic.

Adding capacity

1
2
3
4
5


# Add new host
ceph orch host add ceph-node04 10.0.0.14

# Add OSDs on it
ceph orch daemon add osd ceph-node04:/dev/sdb,/dev/sdc,/dev/sdd

CRUSH will rebalance automatically. Rebalance is expensive — it reshuffles a large fraction of data. For production clusters, add capacity outside business hours and set backfill throttles.

Nearfull and full responses

When a pool approaches full:

ceph osd reweight-by-utilization — slightly lower weight of over-full OSDs, pushing data to less-full ones. Automatic balance.
Add more OSDs.
Delete data.

Once a pool hits full_ratio (default 95%), writes are blocked. Recovering from a full pool requires deleting data or adding OSDs; temporarily raising the full ratio is possible but risky — if you fill it further, you’ll have a full cluster with no room for recovery operations.

Upgrades

cephadm makes upgrades a single command:

1
2


ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.4
ceph orch upgrade status

The orchestrator upgrades daemons in the correct order (MGRs, MONs, OSDs, MDSs, RGWs) with canary deployments and rollback on failure. Don’t skip minor versions during major upgrades — read the release notes and follow the supported upgrade path.

Monitoring

Enable the Prometheus and dashboard modules:

1
2
3


ceph mgr module enable prometheus
ceph mgr module enable dashboard
ceph dashboard set-login-credentials admin 'your-password'

The dashboard at https://<mgr-host>:8443 gives a web view. The Prometheus exporter on <mgr-host>:9283 feeds your existing Prometheus, and the Ceph community ships a Grafana dashboard that covers most operational needs.

Metrics to alert on:

ceph_health_status != 0 (not HEALTH_OK)
ceph_osd_up - ceph_osd_in < 0 (OSDs marked out but still up)
ceph_cluster_total_used_bytes / ceph_cluster_total_bytes > 0.80
ceph_pg_active{state!="active+clean"}
ceph_osd_apply_latency_ms high percentiles
ceph_mon_quorum_status != 1

Common pitfalls

Size=2 replication. Don’t. Use size=3 or erasure coding.
3-node cluster with host failure domain for EC k=4,m=2. Not enough hosts. You need at least 6 (k+m).
Putting OSDs on the OS drive. Crashes. Loss. Sorrow.
Undersized RAM. ceph -s still reports OK until OSDs start OOM-killing under load.
Ignoring nearfull warnings. By the time you hit 95%, rebalance options are limited.
Running without a dedicated cluster_network on busy clusters. Client I/O and replication fight for bandwidth.
Upgrading across too many major versions. Only cross one major version at a time.
Disabling the full-ratio safety to keep writing. You’re two minutes away from a much worse problem.
Using consumer SSDs without PLP (power-loss protection). Ceph issues a lot of fsyncs; consumer SSDs either lose data on power loss or become painfully slow.
No MON backups. The MON store (/var/lib/ceph/<fsid>/mon.*/store.db) contains the cluster maps. Losing all MONs simultaneously is painful; back up the store.

When Ceph is worth it

You need unified block + object + file storage on the same infrastructure.
You have 5+ nodes with 20+ drives and want a single pool of storage.
You need active-active multi-site storage or async DR.
You want to avoid the SAN vendor lock-in story.
You’re comfortable with distributed-systems troubleshooting when things go sideways.

When to pick something else

Small homelabs (1–3 nodes): TrueNAS, Proxmox’s built-in ZFS, or a single MinIO node.
S3-only use cases: standalone MinIO or Garage is far less operational effort.
Under 5 nodes planning to stay that way: Ceph’s overhead is not worth it.
Very small clusters with only HDDs: Ceph performance on spinning disks is painful without SSD DB/WAL devices.

Ceph is not the easy storage system. It’s the reliable storage system. The learning curve is real — plan a weekend to stand up a lab cluster and a week to understand pool design and CRUSH before touching production. Done right, it runs for years, survives hardware failures as a normal operational event, and scales by adding nodes. Done wrong, it’s a collection of racing daemons with confusing error messages. The payoff is proportional to the upfront effort.