Ceph as Software-Defined Storage: Block, File, and Object Across Every Platform

Most storage starts life as a physical thing: a disk, a RAID card, a NAS appliance, a SAN with a four-figure support contract. Software-defined storage (SDS) is the idea that the disk should be dumb and the intelligence should live in software running on commodity servers — replication, healing, placement, snapshots, and the access protocols all implemented in code you can read, on hardware you already own. Ceph is the reference implementation of that idea. It takes a pile of disks across a pile of machines and turns them into one self-healing cluster that speaks block, file, and object simultaneously.

This blog already has two Ceph posts: one on running it hyperconverged under Proxmox, and one on running it standalone with cephadm. This post is deliberately different. It is about consuming the storage — what SDS actually buys you, the three interfaces and when each fits, and the part people struggle with most: getting at the data cleanly from Linux, macOS, and Windows, and wiring it into Proxmox. The networking matters too, but here it is supporting material, not the headline.

All commands and version notes target Ceph Tentacle (20.2.x), the current stable release as of mid-2026, with notes where Squid (19.2.z) differs.

What “software-defined storage” actually means

The phrase gets used loosely, so here is the concrete version. A software-defined storage system gives you four things that a traditional array does not:

• Hardware independence. Any mix of disks on any mix of x86 (or ARM) servers. No proprietary controller, no vendor disk caddies, no forklift upgrade. You add a node, the cluster grows.
• Scale-out, not scale-up. You do not buy a bigger box; you add more boxes, and capacity and throughput grow together. There is no single head unit that becomes the ceiling.
• Policy-driven placement and durability. You declare intent — “keep three copies, never two in the same rack” — and the software decides where every piece of data lives and re-decides automatically when hardware fails.
• Unified, programmable interfaces. The same pool of raw capacity is exposed as block devices, a POSIX filesystem, and an S3 endpoint, all driven by an API.

Ceph delivers all four. The cost is operational complexity — you are now running a distributed system, with all the failure modes that implies. That trade is the recurring theme of this post: Ceph is extraordinarily capable, and it is never the casual choice.

The architecture in one pass

You cannot consume Ceph sensibly without a mental model of what is underneath. Everything in Ceph is built on RADOS (Reliable Autonomic Distributed Object Store), a flat namespace of objects spread across the cluster. The three storage interfaces you actually use are thin translation layers on top of RADOS.

The daemons that make up a cluster:

Two ideas do the heavy lifting:

CRUSH is the placement algorithm. Instead of a central lookup table saying “object X lives on disk Y,” Ceph computes the location from the object name and a CRUSH map that describes your failure domains (disk → host → rack → row). Clients talk directly to the OSD holding their data — there is no metadata bottleneck in the data path. When you change the map (add a host, mark a disk out), CRUSH recomputes placement and the cluster rebalances itself.

Durability comes in two flavors:

• Replication (e.g. size=3) keeps three full copies on three separate failure domains. Simple, fast, and forgiving on recovery — the default for block and file.
• Erasure coding (e.g. k=4, m=2) splits each object into 4 data + 2 parity chunks across 6 domains. You survive 2 failures while storing only 1.5x the data instead of 3x. Cheaper on capacity, heavier on CPU and network, and harder on small random writes — the usual choice for RGW object pools and cold bulk data.

Self-healing falls out of this design: an OSD dies, the MONs notice, CRUSH recomputes, surviving OSDs copy the missing chunks to new homes, and the cluster returns to full redundancy without anyone being paged. That single property — the storage repairs itself — is the strongest argument for SDS over a RAID array that just sits there degraded waiting for you to swap a disk.

The three interfaces, and when each fits

                        ┌───────────────────────────────┐
   Linux / VMs   ─────► │ RBD     (block, krbd / librbd) │
                        ├───────────────────────────────┤
   POSIX clients ─────► │ CephFS  (file, kernel / FUSE)  │ ──► RADOS ──► OSDs (disks)
                        ├───────────────────────────────┤            (CRUSH places data)
   S3 / Swift    ─────► │ RGW     (object, HTTP)         │
                        └───────────────────────────────┘
                        Gateways (SMB / NFS / iSCSI / NVMe-oF)
                        re-export the above to clients that
                        cannot speak Ceph natively.

The rule of thumb: block for one writer, file for many writers, object for applications and bulk. Most real deployments use at least two.

Accessing the data from every platform

This is where homelabbers and small shops get stuck. Ceph’s native clients are excellent on Linux and nearly absent on macOS and Windows. The trick is knowing which gateway re-exports Ceph to the OS in front of you.

Linux — native, and the only first-class citizen

Linux is where Ceph was born and it shows. You get kernel-level clients with no translation layer:

1
2
3
4
5
6
7
8

# RBD: map a block device straight into the kernel
sudo rbd create vmpool/data --size 100G
sudo rbd map vmpool/data            # -> /dev/rbd0
sudo mkfs.xfs /dev/rbd0 && sudo mount /dev/rbd0 /mnt/data

# CephFS: mount the distributed filesystem with the kernel client
sudo mount -t ceph :/ /mnt/cephfs \
  -o name=admin,secretfile=/etc/ceph/admin.secret

For containers, the Ceph CSI driver provisions RBD volumes and CephFS subvolumes dynamically in Kubernetes. There is also rbd-nbd and ceph-fuse for userspace mounts when the kernel version is too old for a feature you need. On Linux, you essentially never need a gateway.

Windows — SMB is the answer, with two new tricks

Windows cannot mount CephFS or map RBD the way Linux does out of the box, so you re-export. There are three real options:

1. SMB via the Ceph SMB Manager module (the modern path). Squid introduced and Tentacle stabilized a first-class ceph smb manager module that deploys clustered Samba (with CTDB for high availability) directly from Ceph, backed by CephFS. You manage it imperatively or declaratively:

   1 2 3 4 5 6

   # Imperative ceph smb cluster create myfiles user --define-user-pass=alice:s3cret ceph smb share create myfiles home cephfs / --subvolume=homes # Declarative — apply a YAML resource description ceph smb apply -i smb-resources.yaml

   It supports both Active Directory domain-member mode and standalone user auth, so Windows clients just see a \\server\share UNC path and never know Ceph is behind it. This is the recommended way to serve Windows in 2026.

2. Native Windows clients (WNBD / Dokan). The Cloudbase-led Windows port ships a ceph-dokan CephFS client and a WNBD-based RBD driver. They work and are genuinely native, but they are a smaller, less-trodden path — fine for a workstation, riskier as the backbone of a fleet.

3. iSCSI or NVMe-oF for block. If a Windows host needs a raw block volume (a cluster shared volume, a SQL Server LUN), export an RBD image over iSCSI or — preferably now — over NVMe/TCP via the Ceph NVMe-oF gateway, which Tentacle improved with gateway groups, multiple namespaces, and sub-10-second failover. Note the older ceph-iscsi project is effectively legacy; NVMe-oF is where the investment is going.

macOS — no native client, so gateway or S3

macOS is the most constrained. There is no native CephFS kernel client and no supported RBD mapping. You have three realistic paths:

• SMB. The same ceph smb clustered-Samba export that serves Windows works perfectly for macOS Finder (Cmd-K → smb://server/share). For a mixed Mac/Windows office this is the single best answer — one gateway, both platforms.
• NFS. Ceph can export CephFS over NFS using clustered nfs-ganesha, managed by ceph nfs cluster create. macOS mounts NFS natively. SMB is usually smoother on modern macOS (better Finder integration, fewer permission surprises), but NFS is there if your environment standardizes on it.
• S3 directly. Anything that speaks S3 — Cyberduck, Mountain Duck (which can present an S3 bucket as a mounted volume), rclone, the AWS CLI, or your own app — talks to RGW from macOS with zero special drivers. For backups and app data this is the cleanest route of all.

The cross-platform summary

The pattern to internalize: Linux speaks Ceph; everyone else speaks a gateway or S3. Plan your cluster with a clustered SMB (and/or NFS) service from the start if you have Macs or Windows machines, and lean on RGW for any workload that can be taught to use object storage.

Proxmox integration

Proxmox is where most people first run Ceph, and the integration is the tightest of any platform — which is exactly why it hides so much that this section is worth reading even if you already clicked through the wizard.

Hyperconverged vs external

There are two ways to pair Proxmox and Ceph:

• Hyperconverged (HCI): Ceph OSDs run on the same nodes as your VMs. Proxmox manages the whole thing through pveceph and the web UI. Simple, dense, and the default homelab pattern. The downside is resource contention — a heavy recovery competes with your VMs for CPU and network — and the coupling of compute and storage scaling.
• External cluster: Ceph runs on its own dedicated nodes (deployed with cephadm), and Proxmox connects to it as a client by dropping in the ceph.conf and keyring under /etc/pve/priv/ceph/. More hardware, cleaner blast radius, independent scaling. The right call once you outgrow the homelab.

Wiring the interfaces into Proxmox

Proxmox consumes two of the three Ceph interfaces directly:

• RBD for VM and container disks. Add an RBD storage pointing at a Ceph pool, and every VM disk becomes a thin-provisioned, snapshottable RBD image that lives in the cluster, not on a local disk. This is what makes live migration and HA possible — any node can reach any VM’s disk because the disk is in Ceph, not under one host.
• CephFS for shared files. Proxmox mounts CephFS for the content types that need to be visible cluster-wide: ISO images, container templates, backup files, and snippets. Upload an ISO once, boot it on any node.

1
2
3

# On a Proxmox node — create the storage definitions
pvesm add rbd ceph-vms --pool vmpool --content images,rootdir
pvesm add cephfs ceph-files --content iso,vztmpl,backup,snippets

RGW is not consumed by Proxmox itself, but you will often run it on the same cluster for backups (point Proxmox Backup Server or your scripts at the S3 endpoint).

The HA payoff

The reason all of this matters: because VM disks live in Ceph and are reachable from every node, Proxmox HA can restart a VM on a surviving node within seconds of a host failure, with no shared-SAN appliance and no data movement. The VM’s disk was never tied to the dead host. Combine three Proxmox+Ceph nodes and you have a self-healing, highly available virtualization cluster on commodity hardware — the open-source answer to a vSphere-plus-SAN stack at a fraction of the cost.

The honest caveat: three nodes is the practical minimum for size=3 replication, and Ceph wants real networking (see below) and real disks (enterprise SSDs with power-loss protection; consumer drives will disappoint you on sync-heavy workloads). Underprovision either and you will blame Ceph for what is really a hardware problem.

Networking: the supporting act that can sink you

Ceph is a distributed system, which means it is a network storage system, and its performance and stability live or die on the network. You do not need to become a network engineer, but you need to get a few things right.

Public vs cluster network. Ceph distinguishes two traffic types:

• Public network — clients talking to OSDs and MONs (the I/O you asked for).
• Cluster network — OSDs talking to each other for replication and recovery (the I/O the cluster generates on its own).

Replication multiplies write traffic: a single client write with size=3 becomes two more writes between OSDs on the cluster network. During recovery after a disk failure, that backend traffic can saturate a link for hours. Putting the cluster network on its own NICs keeps a rebuild from starving client I/O.

1
2
3
4

# ceph.conf
[global]
    public_network  = 10.0.10.0/24
    cluster_network = 10.0.20.0/24

Bandwidth sizing. A rough guide:

Jumbo frames (MTU 9000) measurably help throughput on storage networks, but only if every device in the path agrees — NICs, switches, bonds. A single device stuck at 1500 causes silent fragmentation and maddening intermittent stalls. Set it everywhere or nowhere, and test with ping -M do -s 8972.

For the lowest-latency setups, RDMA over Converged Ethernet (RoCE) can cut OSD-to-OSD latency, but it adds configuration burden (PFC, lossless fabric) that is rarely worth it below a serious production cluster. Start with plain TCP on good NICs.

Pool design, sizing, and the rules that bite

A few decisions made up front save you from pain later:

• Replication vs erasure coding per pool. Use size=3 replication for RBD and CephFS (low-latency, write-heavy). Use erasure coding for RGW and bulk/cold pools where capacity efficiency matters more than write latency. You can mix both in one cluster.
• Placement groups (PGs). PGs are the unit of placement and balancing. Too few and data distributes unevenly; too many and the MONs and OSDs burn memory. Let the PG autoscaler (on by default since Nautilus) handle it unless you have a specific reason not to.
• Failure domains in CRUSH. The default failure domain is host, meaning Ceph will never put two copies of the same data on the same host. If you have racks, set the domain to rack so a rack-level outage cannot take all copies. This is the single most important durability decision and it is one line in a CRUSH rule.
• Never run a cluster full. Ceph needs free space to heal into. The nearfull (85%) and full (95%) thresholds exist for a reason — cross them and the cluster blocks writes to protect itself. Plan capacity so that losing a node still leaves room to re-replicate. In practice, treat ~70% as your comfortable steady-state ceiling.

Day-2 operations, briefly

Running Ceph is mostly leaving it alone, punctuated by a few habits:

• Watch ceph status / ceph health detail. HEALTH_OK is the goal; learn to read HEALTH_WARN (often just a slow scrub or a rebalancing) versus HEALTH_ERR (act now).
• Scrubbing runs automatically to detect bit rot — light scrubs check metadata, deep scrubs re-read data and compare checksums. Let them run; tune their schedule only if they interfere with peak load.
• The balancer module keeps data evenly spread as the cluster changes. It is on by default in upmap mode and mostly invisible.
• Upgrades with cephadm/orchestrator are staged and non-disruptive: ceph orch upgrade start --ceph-version 20.2.1 rolls daemons one at a time. Always upgrade through the supported path (Reef or Squid → Tentacle), never skip two majors.
• Monitoring. The built-in MGR dashboard and Prometheus exporter give you per-OSD latency, PG states, and capacity trends. Alert on nearfull, OSD flapping, and slow ops.

When Ceph is the wrong tool

Honesty matters more than advocacy. Do not reach for Ceph when:

• You have one or two nodes. Ceph’s whole value is distribution. On a single box, ZFS or btrfs gives you better performance, simpler operation, and the same checksumming and snapshots. Use them; this blog has posts on both.
• You need the lowest possible latency for a single workload. A local NVMe drive will always beat a network round-trip. Ceph trades latency for durability, scale, and sharing.
• You will not invest in the networking or the disks. Ceph on 1 GbE with consumer SSDs is a frustrating experience that people then wrongly attribute to Ceph itself.
• You just want a NAS for three people. TrueNAS or a Synology is less to learn and run. Ceph earns its complexity at scale, with multiple clients, or when self-healing distribution is the actual requirement.

Ceph rewards you when you have three or more nodes, real networking, a need to serve block, file, and object from one pool, and the appetite to operate a distributed system. Below that bar, simpler storage wins.

The bottom line

Software-defined storage is the decoupling of storage intelligence from storage hardware, and Ceph is its most complete open-source expression: one self-healing cluster of commodity disks presenting block, file, and object at once. The architecture — RADOS underneath, CRUSH placing data, OSDs healing themselves — is what makes it durable and scalable. The interfaces — RBD, CephFS, RGW — are how you consume it. And the practical reality is that Linux speaks Ceph natively while macOS and Windows reach it through the clustered SMB and NFS gateways or through plain S3, with Proxmox knitting RBD and CephFS into a highly available virtualization platform.

Get the networking right, respect the disk requirements, start at three nodes, and Ceph will quietly run your storage for years. Underprovision any of those and it will teach you, expensively, why the simple options exist. The complexity is real — but so is the payoff of storage that grows when you add a node and fixes itself when one dies.

Sources:

• SMB Meets Squid: Introducing the New Ceph SMB Manager Module
• SMB Service — Ceph Documentation
• v20.2.0 Tentacle released — Ceph.io
• Tentacle — Ceph Documentation