NFS and Network Storage: A Complete Guide from Homelab to the Cloud

March 26, 2026 24 min read Systems Administration

nfsstoragehomelablustrecephiscsisambaazureawslinuxnetworking

Contents

Network storage is how you stop caring which physical machine your data lives on. Whether you’re sharing a media library across three Raspberry Pis, providing shared storage to a Kubernetes cluster, or running a high-performance computing workload that needs parallel I/O across dozens of nodes, the answer is some form of network-attached storage with a protocol that matches the access pattern.

This guide covers the full landscape: the protocols (NFS, SMB, iSCSI), the distributed filesystems (Lustre, Ceph), and how all of it maps to both homelab deployments and cloud services on Azure, AWS, and GCP. By the end you’ll be able to match any storage requirement to the right protocol and deployment model.

The Network Storage Landscape

Before diving into configuration, it helps to understand what category of problem each technology solves:

Technology	Type	Best For
NFSv3	File (NAS)	Unix/Linux file sharing, simple and fast
NFSv4.1/4.2	File (NAS)	Modern Linux, Kubernetes PVs, pNFS parallel I/O
SMB/CIFS (Samba)	File (NAS)	Cross-platform (Windows/Mac/Linux), AD integration
iSCSI	Block (SAN)	VMs, databases — raw block device over TCP
Fibre Channel	Block (SAN)	Enterprise SAN, high throughput, not homelab
Ceph	Object + Block + File	Distributed, scale-out, Kubernetes RWX volumes
Lustre	Parallel File	HPC, large sequential I/O, hundreds of clients
GlusterFS	Distributed File	Scale-out NAS, simpler than Ceph
S3 / Object	Object	Backup, media, unstructured data at scale

NFS: Network File System

NFS is the foundation of Unix network storage. It’s been around since 1984, is supported natively by every Linux distribution and macOS, and remains the dominant protocol for shared Linux filesystems.

NFSv3 vs NFSv4 vs NFSv4.1/4.2

Feature	NFSv3	NFSv4	NFSv4.1	NFSv4.2
Stateful	No	Yes	Yes	Yes
Single port	No (needs portmapper)	Yes (2049)	Yes	Yes
Strong security	No	Yes (Kerberos)	Yes	Yes
File delegation	No	Yes	Yes	Yes
Parallel I/O (pNFS)	No	No	Yes	Yes
Server-side copy	No	No	No	Yes
Sparse files	No	No	No	Yes
Firewall friendly	Difficult	Easy	Easy	Easy

NFSv3 is simpler, stateless, and faster for small deployments. It requires rpcbind and multiple ports (111, 2049, plus dynamic mountd/statd ports), making firewalling awkward.

NFSv4 uses a single TCP port (2049), is stateful with file locking, supports strong security via Kerberos, and is the right choice for any new deployment.

NFSv4.1 with pNFS (parallel NFS) allows clients to read/write directly to multiple storage devices simultaneously — used by enterprise NAS appliances and cloud file services for scale-out performance.

NFSv4.2 adds server-side copy (copy_file_range syscall avoids client→server→client round-trips), sparse file support, and improved security labels.

Setting Up an NFS Server (Linux)

1
2
3
4
5


# Ubuntu / Debian
sudo apt-get install nfs-kernel-server

# RHEL / Fedora
sudo dnf install nfs-utils

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


# /etc/exports syntax:
# <path> <client>(<options>) [<client2>(<options2>)]

# Share to a specific subnet, read-write, with root squash
/tank/media       192.168.1.0/24(rw,sync,no_subtree_check,root_squash)

# Share to a specific host, read-only
/tank/backups     192.168.1.50(ro,sync,no_subtree_check,root_squash)

# Share to multiple specific hosts
/tank/appdata     192.168.1.100(rw,sync,no_subtree_check,no_root_squash) \
                  192.168.1.101(rw,sync,no_subtree_check,no_root_squash)

# Homelab: share to all LAN hosts (less secure, convenient)
/srv/nfs          192.168.1.0/24(rw,sync,no_subtree_check,root_squash)

# NFSv4 pseudo-root (bind mounts into a single tree)
/export           192.168.1.0/24(rw,sync,fsid=0,crossmnt,no_subtree_check)
/export/media     192.168.1.0/24(rw,sync,no_subtree_check,root_squash)
/export/appdata   192.168.1.0/24(rw,sync,no_subtree_check,root_squash)

Key export options explained:

Option	Meaning
`rw` / `ro`	Read-write or read-only
`sync`	Write to disk before acknowledging (safe, slower)
`async`	Acknowledge before disk write (fast, risk of data loss on crash)
`root_squash`	Map remote root to `nobody` (security default — use it)
`no_root_squash`	Allow remote root to act as local root (only for trusted admin hosts)
`all_squash`	Map all users to `nobody` (maximum restriction)
`no_subtree_check`	Don’t verify file is in exported subtree (improves reliability, minor security trade-off)
`fsid=0`	Marks the NFSv4 pseudo-root
`crossmnt`	Allow crossing mount points within the export
`anonuid` / `anongid`	UID/GID to map squashed users to

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# Apply exports
sudo exportfs -ra

# Verify what's exported
sudo exportfs -v

# Enable and start NFS server
sudo systemctl enable --now nfs-kernel-server  # Debian/Ubuntu
sudo systemctl enable --now nfs-server          # RHEL/Fedora

# Check NFS status
sudo systemctl status nfs-kernel-server
rpcinfo -p localhost

NFSv4 Pseudo-Root Setup

NFSv4’s pseudo-root lets you present all shares under a single mount point, avoiding the need for the client to know the exact server path:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Create the export root
sudo mkdir -p /export/media /export/appdata

# Bind mount actual data locations
sudo mount --bind /tank/media /export/media
sudo mount --bind /tank/appdata /export/appdata

# Make bind mounts persistent in /etc/fstab
echo "/tank/media    /export/media    none    bind    0 0" | sudo tee -a /etc/fstab
echo "/tank/appdata  /export/appdata  none    bind    0 0" | sudo tee -a /etc/fstab

1
2
3
4


# /etc/exports for NFSv4 pseudo-root
/export           192.168.1.0/24(rw,sync,fsid=0,crossmnt,no_subtree_check)
/export/media     192.168.1.0/24(rw,sync,no_subtree_check,root_squash)
/export/appdata   192.168.1.0/24(rw,sync,no_subtree_check,root_squash)

Mounting NFS on Clients

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Install NFS client tools
sudo apt-get install nfs-common

# Manual mount (NFSv4, explicit)
sudo mount -t nfs4 -o proto=tcp,port=2049 nas.home.arpa:/export/media /mnt/media

# Auto-detect best version
sudo mount -t nfs nas.home.arpa:/export/media /mnt/media

# Verify mount
mount | grep nfs
df -hT /mnt/media

/etc/fstab — Persistent Mounts

1
2
3
4
5
6
7
8


# NFSv4 mount with recommended options
nas.home.arpa:/export/media   /mnt/media   nfs4   \
  rw,hard,intr,timeo=600,retrans=2,rsize=1048576,wsize=1048576,\
  proto=tcp,nfsvers=4.2,_netdev,auto   0 0

nas.home.arpa:/export/appdata /mnt/appdata nfs4   \
  rw,hard,intr,timeo=600,retrans=2,rsize=1048576,wsize=1048576,\
  proto=tcp,nfsvers=4.2,_netdev,auto   0 0

Mount option reference:

Option	Purpose
`hard`	Retry indefinitely if server unreachable (vs `soft` which returns errors)
`intr`	Allow interrupting hung NFS operations with Ctrl+C
`timeo=600`	Timeout in deciseconds (60s) before retrying
`retrans=2`	Retransmit count before returning an error
`rsize=1048576`	Read block size (1MB — much faster than default 128KB)
`wsize=1048576`	Write block size
`proto=tcp`	Use TCP (default for NFSv4, more reliable than UDP)
`nfsvers=4.2`	Request NFSv4.2 specifically
`_netdev`	Wait for network before mounting
`auto`	Mount at boot
`nofail`	Don’t halt boot if mount fails (useful for optional shares)

Automount with systemd

Automounting mounts the share on first access and unmounts it after a timeout — useful for shares that aren’t always needed:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


# /etc/systemd/system/mnt-media.mount
[Unit]
Description=NAS media share
After=network-online.target
Wants=network-online.target

[Mount]
What=nas.home.arpa:/export/media
Where=/mnt/media
Type=nfs4
Options=rw,hard,intr,rsize=1048576,wsize=1048576,proto=tcp,nfsvers=4.2,_netdev

[Install]
WantedBy=multi-user.target

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# /etc/systemd/system/mnt-media.automount
[Unit]
Description=Automount NAS media share
After=network-online.target
Wants=network-online.target

[Automount]
Where=/mnt/media
TimeoutIdleSec=600   # unmount after 10 minutes idle

[Install]
WantedBy=multi-user.target

1
2
3


sudo systemctl enable --now mnt-media.automount
# Access /mnt/media — triggers automount
# After 10 minutes idle, automatically unmounted

NFS Performance Tuning

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# Check current NFS stats
nfsstat -c    # client stats
nfsstat -s    # server stats
nfsiostat 1   # live I/O statistics (from nfs-utils)

# Increase NFS server threads (default 8 — increase for many clients)
# /etc/nfs.conf:
[nfsd]
threads=32

# Or via /etc/default/nfs-kernel-server (Debian/Ubuntu):
RPCNFSDCOUNT=32

# Tune the kernel NFS read-ahead
echo 15360 | sudo tee /sys/class/bdi/0:*/read_ahead_kb   # 15MB read-ahead

# IRQ affinity for NIC handling NFS traffic
# Find NIC interrupt: cat /proc/interrupts | grep eth0
# Set CPU affinity: echo "ff" > /proc/irq/<N>/smp_affinity

# Server-side: verify rsize/wsize negotiated
sudo cat /proc/fs/nfsd/clients/*/info 2>/dev/null | grep -E "rsize|wsize"

Benchmarking NFS

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


# Install fio for block I/O benchmarks
sudo apt-get install fio

# Sequential read test on NFS mount
fio --name=seqread \
  --directory=/mnt/media \
  --rw=read \
  --bs=1M \
  --size=4G \
  --numjobs=4 \
  --runtime=30 \
  --group_reporting

# Random read test (metadata-heavy, like many small files)
fio --name=randread \
  --directory=/mnt/media \
  --rw=randread \
  --bs=4K \
  --size=512M \
  --numjobs=8 \
  --iodepth=32 \
  --runtime=30 \
  --group_reporting

NFS Security: Kerberos Authentication

By default NFS uses AUTH_SYS — it trusts the client’s claimed UID/GID. Any client on your network can claim to be any user. For a trusted homelab LAN, this is usually fine. For multi-tenant environments, Kerberos (sec=krb5) provides real authentication.

1
2
3
4
5
6
7
8


# On the server — require Kerberos
/export/sensitive  192.168.1.0/24(rw,sync,sec=krb5p,no_subtree_check)
# krb5   = authentication only
# krb5i  = auth + integrity (checksums)
# krb5p  = auth + integrity + privacy (encryption)

# Client mount with Kerberos
sudo mount -t nfs4 -o sec=krb5p nas.home.arpa:/export/sensitive /mnt/sensitive

Setting up a full Kerberos KDC is beyond most homelab needs. For homelab security, combining NFS with a WireGuard VPN tunnel (so only VPN peers can reach the NFS server) achieves similar isolation with much less complexity.

Samba implements the SMB (Server Message Block) protocol, allowing Linux servers to share files with Windows, macOS, and other Linux systems using the same protocol built into every Windows installation.

Installing and Configuring Samba

1
2
3
4
5
6


sudo apt-get install samba

# Create a share directory
sudo mkdir -p /srv/shares/media
sudo chown -R nobody:nogroup /srv/shares/media
sudo chmod 0775 /srv/shares/media

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43


# /etc/samba/smb.conf

[global]
   workgroup = HOMELAB
   server string = Homelab NAS
   server role = standalone server
   log file = /var/log/samba/%m.log
   max log size = 1000
   logging = file
   panic action = /usr/share/samba/panic-action %d

   # Security
   security = user
   map to guest = bad user     # unauthenticated connections get guest access
   usershare allow guests = yes

   # Performance
   socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072
   read raw = yes
   write raw = yes
   use sendfile = yes
   aio read size = 16384
   aio write size = 16384

[media]
   comment = Media Library
   path = /srv/shares/media
   browseable = yes
   writable = yes
   guest ok = yes
   create mask = 0664
   directory mask = 0775
   force group = media

[appdata]
   comment = Application Data
   path = /srv/shares/appdata
   browseable = yes
   writable = yes
   valid users = @samba-users
   create mask = 0660
   directory mask = 0770
   force group = samba-users

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Add a Samba user (must exist as a Linux user first)
sudo useradd -M -s /sbin/nologin smbuser
sudo smbpasswd -a smbuser

# Test config
sudo testparm

# Enable and start
sudo systemctl enable --now smbd nmbd

# Verify shares
smbclient -L localhost -N

Mounting SMB on Linux

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


sudo apt-get install cifs-utils

# Manual mount
sudo mount -t cifs //nas.home.arpa/media /mnt/media \
  -o username=smbuser,password=secret,uid=1000,gid=1000,iocharset=utf8

# Using credentials file (don't put passwords in fstab)
cat > /etc/samba/nas.credentials << EOF
username=smbuser
password=yoursecretpassword
domain=HOMELAB
EOF
chmod 600 /etc/samba/nas.credentials

# /etc/fstab entry
//nas.home.arpa/media  /mnt/media  cifs  \
  credentials=/etc/samba/nas.credentials,uid=1000,gid=1000,\
  iocharset=utf8,vers=3.1.1,_netdev,nofail  0 0

iSCSI: Block Storage Over TCP

Where NFS shares a filesystem, iSCSI shares a raw block device. The client (initiator) sees what looks like a local disk; the OS formats it with any filesystem. This is the right choice for VMs and databases that need a dedicated block device.

Key Concepts

Target: The server exposing the block device
Initiator: The client connecting to and using the block device
LUN (Logical Unit Number): The specific block device being shared
IQN (iSCSI Qualified Name): iqn.2026-01.arpa.home:storage.lun01

Setting Up an iSCSI Target (TGT)

1
2
3
4
5
6


sudo apt-get install tgt

# Create a backing file (or use a real disk/LVM volume)
sudo dd if=/dev/zero of=/var/lib/tgtd/disk01.img bs=1G count=50 status=progress
# Or use a ZFS zvol:
# sudo zfs create -V 50G tank/iscsi/disk01

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


<!-- /etc/tgt/conf.d/iscsi.conf -->
<target iqn.2026-01.arpa.home:storage.lun01>
    # Backing store — file, block device, or ZFS zvol
    backing-store /var/lib/tgtd/disk01.img
    # backing-store /dev/zvol/tank/iscsi/disk01

    # Restrict access to specific initiators
    initiator-address 192.168.1.0/24

    # CHAP authentication (optional)
    incominguser iscsi-initiator secretpassword123
</target>

1
2


sudo systemctl enable --now tgt
sudo tgtadm --mode target --op show

Connecting an iSCSI Initiator (Client)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


sudo apt-get install open-iscsi

# Discover targets on the server
sudo iscsiadm -m discovery -t sendtargets -p 192.168.1.10
# 192.168.1.10:3260,1 iqn.2026-01.arpa.home:storage.lun01

# Login to the target
sudo iscsiadm -m node \
  --targetname iqn.2026-01.arpa.home:storage.lun01 \
  --portal 192.168.1.10:3260 \
  --login

# The disk now appears as /dev/sdb (or similar)
lsblk

# Format and mount
sudo mkfs.ext4 /dev/sdb
sudo mount /dev/sdb /mnt/iscsi

# Persistent login and mount
sudo iscsiadm -m node --targetname iqn.2026-01.arpa.home:storage.lun01 \
  --op update -n node.startup -v automatic

# /etc/fstab with _netdev flag
/dev/disk/by-path/ip-192.168.1.10:3260-iscsi-iqn.2026-01... /mnt/iscsi ext4 \
  _netdev,auto,nofail 0 2

iSCSI Performance Considerations

iSCSI performance is highly sensitive to network quality. For homelab use:

Dedicate a VLAN or a separate NIC to iSCSI traffic
Use jumbo frames (MTU 9000) on the iSCSI network — significant throughput improvement
Enable iSCSI multipath (multipathd) for both redundancy and throughput with multiple NICs

1
2
3
4
5
6


# Enable jumbo frames on the iSCSI interface
sudo ip link set eth1 mtu 9000
# Persist in /etc/netplan or network config

# Verify MTU
ping -M do -s 8972 192.168.1.10   # 8972 + 28 byte header = 9000 MTU test

Lustre: High-Performance Parallel Filesystem

Lustre is the dominant filesystem in high-performance computing (HPC). If you’ve ever used a national supercomputer or a large cloud HPC cluster, you’ve almost certainly used Lustre. It’s designed for one thing: delivering maximum aggregate bandwidth to many clients simultaneously.

Lustre Architecture

┌─────────────────────────────────────────────────────────┐
│  Lustre Filesystem                                       │
├──────────────────┬──────────────────────────────────────┤
│  MGS             │  Management Server                   │
│  (1 node)        │  Stores configuration, startup info  │
├──────────────────┼──────────────────────────────────────┤
│  MDS + MDT       │  Metadata Server + Metadata Target   │
│  (1–many nodes)  │  Directory structure, file metadata  │
├──────────────────┼──────────────────────────────────────┤
│  OSS + OST       │  Object Storage Server + Target      │
│  (many nodes)    │  Actual file data, striped across    │
│                  │  multiple OSTs for parallel I/O      │
├──────────────────┼──────────────────────────────────────┤
│  Clients         │  Mount lustre:/ using Lustre client  │
│  (thousands)     │  Each file striped across N OSTs     │
└──────────────────┴──────────────────────────────────────┘

A file in Lustre is striped across multiple OSTs. Reading a 10 GB file with stripe count 8 means 8 OSS nodes each serve 1.25 GB simultaneously — aggregate read bandwidth scales with the number of OSTs.

Lustre in a Homelab

Running full production Lustre requires at least 4–5 nodes (MGS, MDS, 2+ OSS, clients). That’s more than most homelabs have, but a minimal single-node Lustre setup is useful for learning and small experiments.

Minimum homelab Lustre cluster:

1 node: MGS + MDS + 2 OSTs (can all be on one server for testing)
1+ client nodes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# Install Lustre (RHEL/Rocky — Lustre has best RHEL support)
# Download from lustre.org or whamcloud.com

# On the combined MGS/MDS/OSS node:
# Format the MGS
mkfs.lustre --mgs /dev/sdb

# Format the MDT
mkfs.lustre --mdt --fsname=homelab --mgsnode=192.168.1.10@tcp --index=0 /dev/sdc

# Format OSTs
mkfs.lustre --ost --fsname=homelab --mgsnode=192.168.1.10@tcp --index=0 /dev/sdd
mkfs.lustre --ost --fsname=homelab --mgsnode=192.168.1.10@tcp --index=1 /dev/sde

# Mount them
mount -t lustre /dev/sdb /mnt/mgs
mount -t lustre /dev/sdc /mnt/mdt
mount -t lustre /dev/sdd /mnt/ost0
mount -t lustre /dev/sde /mnt/ost1

# On the client:
mount -t lustre 192.168.1.10@tcp:/homelab /mnt/lustre

Lustre Striping

The power of Lustre is striping. Set the stripe count based on your access pattern:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


# Set default stripe for a directory (new files inherit this)
lfs setstripe -c 4 -S 4M /mnt/lustre/large_files
# -c 4 = stripe across 4 OSTs
# -S 4M = 4MB stripe size

# Check stripe for a file
lfs getstripe /mnt/lustre/large_files/bigdata.tar
# lmm_stripe_count:  4
# lmm_stripe_size:   4194304
# lmm_layout_gen:    0
# lmm_stripe_offset: 1
#   obdidx  objid  group  fid
#      1    12345  0      ...
#      2    12346  0      ...
#      3    12347  0      ...
#      0    12348  0      ...

# Optimal stripe count = number of OSTs for maximum bandwidth
# For small files (< 1MB): stripe count 1 (no benefit from striping)
# For large files (> 100MB): stripe count = all OSTs
# For mixed: stripe count 2-4

# Check filesystem status
lfs df -h /mnt/lustre
lctl dl    # list Lustre devices

Lustre for ML/AI Workloads

Lustre is the storage choice for large-scale machine learning training — reading hundreds of gigabytes of training data at maximum speed. The key tunables:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Maximize read performance for sequential large-file access
lfs setstripe -c -1 /mnt/lustre/training_data  # stripe across ALL OSTs

# Increase client read-ahead
lctl set_param llite.*.max_read_ahead_mb=256
lctl set_param llite.*.max_read_ahead_per_file_mb=256

# Parallel read with multiple threads
lfs find /mnt/lustre/training_data -type f | \
  xargs -P 16 -I{} cp {} /dev/null   # stress test parallel read

Lustre in the Cloud

AWS FSx for Lustre: Fully managed Lustre on AWS. Integrated with S3 — you can transparently read S3 objects as Lustre files. Ideal for ML training jobs that read from S3.

1
2
3
4
5


# Mount FSx for Lustre on EC2
sudo amazon-linux-extras install lustre
sudo mount -t lustre -o noatime,flock \
  fs-0abcdef123456789.fsx.us-east-1.amazonaws.com@tcp:/fsx \
  /mnt/fsx

Azure HPC Cache: Microsoft’s managed NFS/Lustre caching layer for HPC workloads on Azure. Sits in front of Azure Blob Storage or on-premises NFS servers and presents a cached NFS mount to compute nodes.

Google Cloud Lustre (via DDN, WekaFS): GCP doesn’t offer native managed Lustre but partners like DDN (EXAScaler) and WekaFS provide high-performance parallel filesystems on GCP for HPC and AI workloads.

Ceph: The Software-Defined Storage Platform

Ceph is the Swiss Army knife of distributed storage. It provides object storage (S3-compatible), block storage (RBD), and a POSIX filesystem (CephFS) from a single storage cluster. It’s designed to scale from a few nodes to thousands, with no single point of failure.

Ceph Architecture

┌─────────────────────────────────────────────────┐
│  Ceph Cluster                                   │
├───────────────┬─────────────────────────────────┤
│  MON nodes    │  Monitors — cluster map, quorum │
│  (3 or 5)     │  (like etcd for Ceph)           │
├───────────────┼─────────────────────────────────┤
│  MGR nodes    │  Managers — metrics, dashboard  │
├───────────────┼─────────────────────────────────┤
│  OSD nodes    │  Object Storage Daemons         │
│  (1 per disk) │  Store actual data, replication │
├───────────────┼─────────────────────────────────┤
│  MDS nodes    │  Metadata Servers (CephFS only) │
└───────────────┴─────────────────────────────────┘

Access methods:
  RADOS Gateway (RGW) → S3-compatible object storage
  RBD (RADOS Block Device) → Block storage for VMs
  CephFS → POSIX filesystem (NFS-like, usable as Kubernetes RWX PV)

Homelab Ceph with Rook

Rook is the most practical way to run Ceph in a homelab Kubernetes cluster — it deploys and manages Ceph as Kubernetes operators:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


# Install Rook operator
helm repo add rook-release https://charts.rook.io/release
helm install --create-namespace --namespace rook-ceph rook-ceph rook-release/rook-ceph

# Create a CephCluster (3 nodes, 1 OSD per node)
# rook-cluster.yaml
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v18
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    count: 1
  storage:
    useAllNodes: true
    useAllDevices: false
    devices:
    - name: "sdb"   # dedicated disk on each node
  network:
    provider: host   # use host network for better performance

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# Create a storage class for RBD (block storage for RWO PVCs)
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  replicated:
    size: 3
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: replicapool
  imageFormat: "2"
  imageFeatures: layering
reclaimPolicy: Retain
allowVolumeExpansion: true

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# Create a storage class for CephFS (shared RWX PVCs)
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: myfs
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
  dataPools:
  - name: data0
    replicated:
      size: 3
  metadataServer:
    activeCount: 1
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-cephfs
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph
  fsName: myfs
  pool: myfs-data0
reclaimPolicy: Retain
allowVolumeExpansion: true

With these storage classes, Kubernetes pods can claim RWO (ReadWriteOnce) block volumes via rook-ceph-block and RWX (ReadWriteMany) shared volumes via rook-cephfs — the latter allows multiple pods on different nodes to share the same filesystem.

Cloud Network Storage: Azure, AWS, and GCP

Each cloud provider offers managed equivalents of the storage protocols above. Understanding the mapping helps you choose the right service.

Azure File Storage Options

Azure Files

Managed SMB and NFS file shares backed by Azure Blob Storage:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


# Mount Azure Files (SMB) on Linux
sudo apt-get install cifs-utils

# Get storage account key from Azure Portal or CLI
STORAGE_ACCOUNT="mystorageaccount"
SHARE_NAME="myshare"
STORAGE_KEY="base64key=="

sudo mount -t cifs //${STORAGE_ACCOUNT}.file.core.windows.net/${SHARE_NAME} \
  /mnt/azure \
  -o username=${STORAGE_ACCOUNT},password=${STORAGE_KEY},\
     serverino,nosharesock,actimeo=30,mfsymlinks

# Mount Azure Files (NFS) — requires Premium tier and VNet integration
sudo mount -t nfs4 \
  ${STORAGE_ACCOUNT}.file.core.windows.net:/${STORAGE_ACCOUNT}/${SHARE_NAME} \
  /mnt/azure-nfs \
  -o vers=4,minorversion=1,sec=sys

Azure Files tiers:

Transaction Optimized: Standard HDD, cheap, high-latency — good for cold storage and backups
Hot / Cool: Standard SSD-backed — general purpose file shares
Premium: NVMe-backed, <1ms latency — databases, VMs, low-latency apps

Azure NetApp Files (ANF)

ANF is enterprise-grade NFS/SMB for latency-sensitive and high-throughput workloads. It’s based on NetApp ONTAP technology running natively in Azure datacenters.

1
2
3
4
5
6
7


# ANF presents as a standard NFS mount
sudo mount -t nfs -o rw,hard,rsize=65536,wsize=65536,vers=3,tcp \
  10.0.1.4:/volume-name /mnt/anf

# Or NFSv4.1
sudo mount -t nfs4 -o rw,hard,rsize=65536,wsize=65536,vers=4.1,tcp \
  10.0.1.4:/volume-name /mnt/anf

ANF features relevant to enterprise homelab overflow:

Snapshot policies: Automated hourly/daily/weekly snapshots with retention
Cross-region replication: Async replication of volumes to a paired region
Capacity pools: Pool of provisioned capacity; volumes carved from the pool
Performance tiers: Standard (16 MiB/s per TiB), Premium (64 MiB/s per TiB), Ultra (128 MiB/s per TiB)

ANF pricing is high (~$0.245/GiB/month for Premium vs ~$0.06/GiB for Azure Files Premium) but justified for workloads needing Oracle, SAP HANA, or HPC performance.

Azure HPC Cache

For HPC workloads on Azure that read from on-premises NFS or Azure Blob Storage, Azure HPC Cache acts as a caching tier — presenting a fast NFS mount to compute VMs that caches data from slower backends:

On-premises NFS / Azure Blob → Azure HPC Cache → VM Compute Nodes (NFS mount)

The cache is sized in TiB (3–48 TiB) and handles cache invalidation, write-back, and namespace aggregation transparently.

AWS File Storage Options

Amazon EFS (Elastic File System)

EFS is fully managed NFSv4 — you mount it like any NFS server, it scales automatically, and you pay per GB stored:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


# Install EFS mount helper (recommended over raw NFS mount)
sudo apt-get install amazon-efs-utils

# Mount with EFS mount helper (handles TLS, stunnel, retries)
sudo mount -t efs fs-0abcdef12345678:/ /mnt/efs

# With TLS encryption
sudo mount -t efs -o tls fs-0abcdef12345678:/ /mnt/efs

# /etc/fstab
fs-0abcdef12345678.efs.us-east-1.amazonaws.com:/ /mnt/efs efs \
  _netdev,tls,iam,nofail 0 0

EFS performance modes:

General Purpose: Low latency, up to 35K IOPS — most workloads
Max I/O: Higher aggregate throughput, slightly higher latency — parallel workloads with many clients
Elastic throughput (new default): Automatically scales to workload, no provisioning needed
Provisioned throughput: Set a specific throughput baseline for predictable workloads

Amazon FSx Family

AWS offers four managed FSx variants:

FSx Type	Protocol	Best For
FSx for Windows	SMB	Windows workloads, AD-integrated
FSx for Lustre	Lustre	HPC, ML training, S3 integration
FSx for NetApp ONTAP	NFS + SMB + iSCSI	Multi-protocol, snapshots, tiering
FSx for OpenZFS	NFS	ZFS snapshots, clones, compression

FSx for Lustre is particularly powerful for ML workloads — it can lazily import data from S3, so the first read of a file pulls it from S3 and caches it on the Lustre filesystem:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


# Mount FSx for Lustre
sudo yum install -y lustre-client
sudo mount -t lustre -o noatime,flock \
  fs-0abc123def456789.fsx.us-east-1.amazonaws.com@tcp:/fsx \
  /mnt/fsx

# HSM (Hierarchical Storage Management) — export back to S3
sudo lfs hsm_archive /mnt/fsx/results/output.tar   # archive to S3
sudo lfs hsm_release /mnt/fsx/dataset/old_data/     # free local space
sudo lfs hsm_restore /mnt/fsx/dataset/old_data/     # restore from S3

GCP File Storage Options

Google Cloud Filestore

Managed NFS — similar to EFS but for GCP:

1
2
3
4
5
6
7


# Mount Filestore from a GCE instance
sudo apt-get install nfs-common
sudo mount -t nfs -o rw,hard,intr,timeo=600 \
  10.0.0.2:/vol1 /mnt/filestore

# /etc/fstab
10.0.0.2:/vol1 /mnt/filestore nfs rw,hard,intr,timeo=600,_netdev 0 0

Filestore tiers:

Basic HDD: $0.20/GiB/month, 100 MiB/s, 600 IOPS
Basic SSD: $0.30/GiB/month, 480 MiB/s, 25K IOPS
High Scale SSD: Petabyte-scale, 10 GiB/s+, Lustre-like performance
Enterprise: 99.99% SLA, multi-zone redundancy, snapshots

Kubernetes Persistent Volumes: Tying It Together

All of these storage options surface in Kubernetes as PersistentVolumes. Here’s how to use them:

NFS PersistentVolume (Static)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-media-pv
spec:
  capacity:
    storage: 2Ti
  accessModes: [ReadWriteMany]   # NFS supports multiple readers/writers
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: nas.home.arpa
    path: /export/media
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: media-pvc
  namespace: jellyfin
spec:
  accessModes: [ReadWriteMany]
  resources:
    requests:
      storage: 2Ti
  volumeName: nfs-media-pv

Dynamic NFS Provisioning with nfs-subdir-external-provisioner

1
2
3
4
5
6
7
8
9


helm repo add nfs-subdir-external-provisioner \
  https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

helm install nfs-provisioner \
  nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
  --set nfs.server=nas.home.arpa \
  --set nfs.path=/export/k8s \
  --set storageClass.name=nfs-client \
  --set storageClass.reclaimPolicy=Retain

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# Now any PVC can use NFS dynamically
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-data
spec:
  accessModes: [ReadWriteMany]
  storageClassName: nfs-client
  resources:
    requests:
      storage: 10Gi

CSI Drivers for Cloud Storage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


# EFS on EKS
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-0abc123def
  directoryPerms: "700"

# Azure Files on AKS
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azure-files-premium
provisioner: file.csi.azure.com
parameters:
  skuName: Premium_LRS
  storageAccount: mystorageaccount
reclaimPolicy: Retain
allowVolumeExpansion: true

Choosing the Right Protocol: A Decision Framework

What's the primary access pattern?
├── File access (open, read, write, seek)
│   ├── Linux/Unix clients only?
│   │   ├── Single server → NFSv4.2
│   │   ├── Kubernetes RWX volumes → CephFS or NFS
│   │   └── HPC / parallel I/O → Lustre (or FSx for Lustre on AWS)
│   └── Mixed Linux + Windows + macOS?
│       └── Samba (SMB 3.x)
│
├── Block access (raw disk, VM storage, databases)
│   ├── Local (same server) → LVM, ZFS zvol
│   ├── Network SAN → iSCSI
│   └── Cloud / Kubernetes RWO → Ceph RBD, EBS, Azure Disk
│
└── Object access (S3 API, unstructured data)
    ├── Self-hosted → MinIO, Ceph RGW
    ├── AWS → S3
    ├── Azure → Azure Blob Storage
    └── GCP → Google Cloud Storage

Homelab Reference Architecture

A practical homelab storage stack combining multiple protocols:

NAS Server (ZFS pool: tank, raidz2, 4× 4TB HDDs)
├── NFS Server
│   ├── /export/media     → All Pis and K3s nodes (media apps)
│   ├── /export/appdata   → K3s nfs-provisioner (dynamic PVCs)
│   └── /export/k8s       → Longhorn fallback, Kubernetes static PVs
│
├── Samba Server
│   ├── \\nas\media       → Windows/macOS clients
│   └── \\nas\backups     → Time Machine target (macOS)
│
├── iSCSI Target
│   └── LUN backed by ZFS zvol → Proxmox VMs (high-performance VM disks)
│
└── Litestream
    └── SQLite WAL replication → Hetzner Object Storage (off-site)

K3s Cluster (Rook/Ceph on NVMe SSDs)
├── rook-ceph-block   → RWO PVCs (databases, single-writer apps)
└── rook-cephfs       → RWX PVCs (shared config, media mounts)

Cloud Overflow (Hetzner)
└── Hetzner Object Storage → Restic backups, Litestream, static assets

Network storage complexity scales with your needs. Start with a simple NFS share from a Linux box or NAS appliance — that covers 90% of homelab use cases. Add iSCSI when you need raw block devices for VMs. Add Ceph (via Rook) when you need distributed storage that survives node failures. Reach for Lustre only when you have genuine high-performance parallel I/O requirements and multiple storage nodes to dedicate to it.

The cloud equivalents mirror this ladder exactly: EFS/Azure Files for shared file storage, EBS/Azure Disk for block, and FSx for Lustre when you need HPC-grade parallel I/O. Knowing the on-premises version makes the cloud version trivial to understand and operate.