LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

Rook: Ceph as a Kubernetes Operator

14 min read Virtualization & Containers
rookcephkubernetesstorageoperatorinfrastructure
Contents

There is a version of Ceph that you install on bare metal, wire up by hand, and spend a non-trivial fraction of your life keeping running. That version is real, it is powerful, and if you want the unvarnished guide to it, the standalone-ceph-without-proxmox post covers it. This post is about a different bet: what happens when you stop treating Ceph as an external dependency and let a Kubernetes operator own the whole thing — deploying it, wiring it to storage classes, and handling day-2 operations through the same reconciliation loop that runs your application workloads.

That operator is Rook. It graduated from the CNCF in October 2020 and is, at this writing, the canonical way to run Ceph inside Kubernetes. The version axis matters for clarity: Rook is the operator and orchestration layer; Ceph is the actual distributed storage system doing the work. Rook does not store your data — Ceph does. Rook just makes sure the right Ceph daemons are running as Kubernetes pods, that the right PersistentVolumes get provisioned when you ask for them, and that when a disk fails or a node reboots the cluster heals without your pager going off.

What an operator actually does here

The kubernetes-operators-deep-dive post covers the operator pattern in full. The short version for Rook: you install the Rook operator as a Deployment, it registers a set of Custom Resource Definitions, and from that point on you manage your storage cluster by writing YAML — specifically, by creating CephCluster, CephBlockPool, CephFilesystem, and CephObjectStore resources. The operator watches those resources, computes the difference between what you declared and what is running, and reconciles.

In practice that means the operator translates your CephCluster spec into a running ensemble of Ceph daemons as Kubernetes pods:

   CephCluster CR
        │
        ▼
   Rook Operator (Deployment)
        │
        ├── MON pods  (3 or 5: quorum, cluster map, failover elections)
        ├── MGR pods  (dashboard, Prometheus metrics, cephadm orchestrator shim)
        ├── OSD pods  (one per disk or PVC, the actual data layer)
        ├── MDS pods  (CephFS metadata servers, one active + one standby minimum)
        └── RGW pods  (RADOS Gateway, the S3/Swift-compatible object endpoint)

Every one of those is a regular Kubernetes pod. They can be inspected with kubectl logs, scheduled with NodeAffinity and Tolerations, and — critically — reconciled automatically when they crash or drift. The operator owns the lifecycle end to end: if you add a new disk by labeling a node, it creates a new OSD pod. If you change the CephCluster spec to request five MONs instead of three, the operator brings up two more. If an OSD pod dies because its underlying disk failed, the operator marks the OSD out and starts the rebalance without you doing anything.

This is what distinguishes Rook from “install Ceph on your Kubernetes nodes by hand and hope.” The operator model means the cluster is continuously reconciled toward your declared intent, with Kubernetes handling the pod scheduling and health checking that you would otherwise write yourself.

The CephCluster CRD

Everything starts with a CephCluster. This is the top-level object that tells Rook where your storage nodes are, which disks to use, and how to tune the basic cluster shape:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v20          # Tentacle (20.x) as of mid-2026
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false           # never in production
  mgr:
    count: 2
    modules:
      - name: pg_autoscaler
        enabled: true
  dashboard:
    enabled: true
    ssl: true
  storage:
    useAllNodes: false                    # explicit is safer
    useAllDevices: false
    nodes:
      - name: "worker-1"
        devices:
          - name: "nvme0n1"
      - name: "worker-2"
        devices:
          - name: "nvme0n1"
      - name: "worker-3"
        devices:
          - name: "nvme0n1"
  resources:
    osd:
      limits:
        memory: "4Gi"
      requests:
        memory: "2Gi"

A few choices here that matter in production. allowMultiplePerNode: false is the setting that prevents two MON pods from landing on the same node, which would defeat the purpose of having three. useAllDevices: false paired with explicit device lists is how you make sure Rook does not accidentally claim a disk you were planning to use for something else. The cephVersion.image tag pins Ceph; in mid-2026 that means Ceph 20 (Tentacle), which ships Crimson-OSD improvements and fast-EC, or Ceph 19 (Squid, EOL estimated September 2026) if your environment requires the older LTS.

Block storage: CephBlockPool and StorageClass

For ReadWriteOnce (RWO) block storage — what most stateful workloads want — you define a CephBlockPool and wire it to a StorageClass. The pool sets the replication policy; the StorageClass is what application developers actually reference in their PVCs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host        # one replica per host, not per OSD
  replicated:
    size: 3
    requireSafeReplicaSize: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: replicapool
  imageFormat: "2"
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
reclaimPolicy: Delete
allowVolumeExpansion: true

With this in place, a stateful application requests storage the normal Kubernetes way:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rook-ceph-block
  resources:
    requests:
      storage: 50Gi

The Rook CSI driver creates a RADOS Block Device (RBD) image in the pool, maps it into the pod as a block device, and manages the entire lifecycle including resize and snapshot. This is the most common Rook-Ceph usage pattern — it is the right default for databases, message queues, and any workload that writes to a single node.

Shared filesystem: CephFilesystem and ReadWriteMany

When multiple pods need to read and write the same volume simultaneously — a shared build cache, media assets, config that several replicas must consume — RBD cannot help because an RBD image is single-writer by design. CephFS fills that gap:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: myfs
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
  dataPools:
    - name: default
      failureDomain: host
      replicated:
        size: 3
  preserveFilesystemOnDelete: true   # do not wipe data when the CRD is deleted
  metadataServer:
    activeCount: 1
    activeStandby: true              # hot-standby MDS, no failover delay
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-cephfs
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph
  fsName: myfs
  pool: myfs-data0
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
reclaimPolicy: Delete
allowVolumeExpansion: true

PVCs from this StorageClass can use accessModes: [ReadWriteMany], giving you a POSIX filesystem mounted by many pods simultaneously. The honest caveat: CephFS RWX workloads need some care with metadata-heavy access patterns (many small files, lots of stat calls), because all metadata operations funnel through the MDS. For large sequential writes or reads, CephFS is excellent. For workloads that generate many tiny files, benchmark before committing.

Object storage: CephObjectStore and the S3 endpoint

Rook can also expose an S3-compatible endpoint via the RADOS Gateway (RGW), which means your in-cluster applications can talk to what looks exactly like AWS S3 without a cloud account:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: my-store
  namespace: rook-ceph
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPool:
    failureDomain: host
    erasureCoded:
      dataChunks: 2
      codingChunks: 1
  preservePoolsOnDelete: true
  gateway:
    port: 80
    instances: 2              # two RGW pods for HA
  healthCheck:
    bucket:
      disabled: false
      interval: 60s

Rook creates a Kubernetes Service for the RGW endpoint. Application code that talks to MinIO or AWS S3 (using the AWS SDK with a custom endpoint) works unchanged — you point it at the RGW Service URL instead of s3.amazonaws.com. This is particularly useful for backup tooling, artifact storage, and ML data pipelines that need durable object storage without leaving the cluster or incurring egress costs.

Day-2 operations

The genuine test of an operator is what happens after the initial install. Rook holds up well on day-2 work.

Adding OSDs. Attach a new disk to a worker node and add it to the CephCluster spec. The operator detects the new device, creates a job to prepare it (format, create OSD directory), and brings up a new OSD pod. Ceph then automatically rebalances data across the expanded OSD set. No manual ceph osd create needed.

Upgrading Ceph. Change the cephVersion.image field in the CephCluster spec:

1
2

kubectl -n rook-ceph patch cephcluster rook-ceph --type merge \
  -p '{"spec":{"cephVersion":{"image":"quay.io/ceph/ceph:v20.2.0"}}}'

The operator performs a rolling upgrade: MGR first, then MONs, then OSDs one at a time, waiting for health HEALTH_OK at each step before proceeding. It is not instant — a large cluster can take an hour — but it is supervised and will pause automatically if the cluster enters a degraded state.

The toolbox pod. When something is wrong and you need the raw Ceph CLI, the toolbox pod is your friend:

1
2
3
4
5
6
7
8

kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

# inside the toolbox
ceph status
ceph osd tree
ceph df
ceph health detail
ceph osd pool ls detail

The toolbox deploys with the Ceph client binaries and the cluster credentials already mounted, so you get a fully functional ceph command without installing anything on your nodes. For any cluster health investigation — checking placement group states, finding slow OSDs, diagnosing network issues between daemons — this is the first stop.

Disk failure. When a disk dies, the OSD pod will crash and the operator will not restart it (a failed OSD that is crashlooping would make things worse). The OSD lands in down state, Ceph marks it out after the configured timeout (default 10 minutes), and rebalancing starts automatically. You then identify the failed OSD, remove it from the cluster spec, and replace the physical disk. The operator handles the pod cleanup.

External cluster mode

Not every team wants to run Ceph daemons inside Kubernetes. You might already have a well-managed Ceph cluster (perhaps managed by Proxmox Ceph or cephadm), and you want Kubernetes workloads to consume it via RBD and CephFS without re-running the entire daemon set in-cluster.

That is exactly what Rook’s external mode does. You run the Rook operator and CSI driver in Kubernetes, but point them at an external Ceph cluster instead of managing one internally. The CephCluster spec reflects this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph-external
  namespace: rook-ceph-external
spec:
  external:
    enable: true
  dataDirHostPath: /var/lib/rook
  cephVersion:
    image: quay.io/ceph/ceph:v20

You then run Rook’s import script against the external cluster to extract credentials and connection details, creating the necessary Secrets in the Kubernetes namespace. From that point, CephBlockPool and CephFilesystem CRDs reference pools that already exist on the external cluster, and the Rook CSI driver provisions PVCs against them. The MON, MGR, and OSD pods are all absent from the Kubernetes cluster — they are running elsewhere.

   External Ceph Cluster           Kubernetes Cluster
   (cephadm / Proxmox)             (Rook external mode)
   ─────────────────────           ────────────────────
   MON × 3                         Rook Operator
   MGR × 2            ◀────────▶   CSI Driver (RBD + CephFS)
   OSD × N                         CephBlockPool CR (ref only)
   MDS, RGW                        StorageClasses
                                    PVCs → PVs provisioned via CSI

External mode is the right choice when you want Rook’s provisioning and CSI integration without surrendering control of the Ceph cluster itself, or when the Ceph cluster is shared across workloads beyond Kubernetes (NFS exports, S3 access from outside the cluster, direct RBD mounts on bare-metal VMs).

Rook-Ceph vs Longhorn vs cloud block storage

Rook-Ceph Longhorn Cloud block (EBS/GCE PD)
Access modes RWO, RWX (CephFS), object S3 RWO only (RWX via NFS, awkward) RWO
Complexity High — Ceph is large Low — single Helm chart Near-zero
Resource overhead Significant (3+ MON, 1+ MGR, OSDs) Low per-node agents None (managed)
Performance Excellent for IOPS-heavy workloads Good for general purpose Vendor-dependent
Upgrade path Operator-managed rolling upgrade Simple Helm upgrade Transparent
Works on bare metal Yes, primary use case Yes No
External cluster mode Yes No N/A
Snapshots / clones Native Ceph snapshots via CSI Yes Yes
CNCF status Graduated Incubating N/A

Longhorn is a SUSE Rancher project that replicates block volumes across nodes by copying raw disk data, rather than using a distributed storage protocol like RADOS. It installs in minutes with a Helm chart, has a polished dashboard, and is genuinely excellent for clusters of three to ten nodes where you want durable RWO storage without the operational surface area of Ceph. If block storage is all you need and you are not running at scale, Longhorn is the right tool. The trade-off is narrow: no RWX, no object storage, no external cluster mode, and replication is data-level rather than protocol-level — meaning performance under heavy write loads is bounded by the slowest replica’s throughput.

Cloud-managed block storage (EBS, GCE Persistent Disk, Azure Disk) is the obvious answer on cloud-hosted clusters where the control plane is already managed. If you are on EKS or GKE there is almost no reason to run Rook — you are paying for managed storage anyway, it works without operator overhead, and its reliability is someone else’s problem. Rook makes the most sense on bare metal, in homelabs, at the edge, and in private cloud environments where cloud-native storage primitives are not available.

When Rook-Ceph is overkill

Ceph is a large system. The minimum viable production Rook-Ceph cluster — three nodes, three MONs, three OSDs, an MGR, a CSI driver — consumes something in the range of 4–8 GB of RAM for the Ceph daemons alone, before your workloads get a byte. On a three-node homelab running 8 GB nodes, Rook is eating a third of your memory before your applications start. That is not a bug; it is an honest consequence of what Ceph is: a distributed system designed to tolerate failure at scale, with the machinery to match.

The honest “do not use Rook” checklist:

  • Fewer than three worker nodes. Ceph requires three MONs for quorum. A two-node cluster cannot form a healthy Ceph cluster, and a single-node cluster is a contradiction in terms for a system designed around replication across failure domains.
  • Nodes with less than 8–16 GB RAM. The OSD daemons are memory-hungry; the default OSD memory target is 4 GB per OSD. On thin nodes you will spend your time tuning memory limits and watching OSD pods get OOM-killed.
  • Block-only workloads at small scale. If you need RWO PVCs and nothing else, Longhorn installs in ten minutes and does not require understanding placement groups.
  • Cloud-hosted clusters. EBS and GCE PD exist for a reason. Use them.
  • Teams without Ceph operational experience. Rook reduces Ceph operations dramatically but does not eliminate them. Degraded pools, OSD failures, and networking issues between Ceph daemons still require someone who knows what ceph health detail is telling them. The toolbox pod is powerful — but only if you know what you are reading.

The flip side of that list is equally real: for bare-metal clusters at medium-to-large scale, for environments that need RWX and object storage alongside block, or for teams that want a single storage system powering multiple access patterns, Rook-Ceph is the most capable option available in the cloud-native ecosystem.

The verdict

Rook earns its CNCF Graduated status by doing something genuinely hard: taking Ceph — a system that has historically required dedicated administrators and careful hand-holding — and making it operate like a Kubernetes workload. The operator reconciliation loop handles the day-2 work that burned people out on standalone Ceph. The CSI integration means developers write a StorageClassName and get block, filesystem, or object storage without knowing a placement group from a pool. External cluster mode lets you bridge a cephadm-managed cluster into Kubernetes without re-running the daemon set.

The prerequisite is honest: you need a cluster with the resources to run Ceph comfortably, at least three nodes, and someone on the team who can read ceph status when things go sideways. For a two-node homelab, Longhorn. For a cloud cluster with managed storage, use the cloud. But for a substantial bare-metal Kubernetes environment that needs durable, high-performance, multi-modal storage — block for your databases, CephFS for your shared workloads, S3-compatible object storage for your pipelines — Rook-Ceph is the answer, and running it as an operator is the right way to do it.

Sources

Comments