LUNAROPS · OPERATIONAL UPLINK 100% UPTIME 1,247d POSTS 893 JEFF.MOON@LUNAROPS.DEV UTC --:--:--

NVMe-over-Fabrics: Block Storage at Near-Local Latency Over the Network

nvme-ofstoragenetworkingrdmalinuxinfrastructureperformance

The single biggest reason iSCSI feels slow is not the network — it is that SCSI is a single-queue, command-tag-serialized protocol bolted to a transport designed for spinning disks. Every command goes through one queue with at most a handful of outstanding tags, every block read pays a SCSI-to-NVMe translation tax on the target, and every operation drags a 1980s session-layer state machine along for the ride. NVMe-over-Fabrics throws all of that out. It takes the same multi-queue, lockless submission/completion model that PCIe NVMe uses on a local bus and runs it over a network, so a remote SSD looks to the host like a local one with maybe 30 extra microseconds of latency.

The result is the first network storage protocol that does not feel like a downgrade. A single NVMe-oF connection can drive a 25GbE link to wire speed against a single SSD without saturating one CPU core, sustain 80–90% of local NVMe IOPS for random reads, and add only tens of microseconds of latency when the fabric is decent. The price is a sharper edge: NVMe-oF assumes a network that behaves, transports that don’t all interoperate, multipath behavior that is fundamentally different from dm-multipath, and a discovery story that for years was the worst part of the whole stack.


The queue model is the protocol

To understand NVMe-oF you have to understand what NVMe actually is. It is not a wire format — it is a queue-based register interface. The host creates a Submission Queue (SQ) and a Completion Queue (CQ) in host memory, writes a command into the SQ, rings a doorbell, and the controller posts a completion into the CQ when it’s done. Each I/O is a 64-byte command and a 16-byte completion. There is no session, no SCSI CDB, no LUN reset state machine — just queues.

The PCIe transport binds those queues to MMIO doorbells and DMA. The fabrics transports bind the same queues to a network. The command set is identical. The on-the-wire encoding of a command and completion is identical at the byte level. The only thing that changes is how the bytes get from host memory to controller memory and back.

                NVMe abstraction model
                =======================

  Host                                       Controller
  ----                                       ----------
  +------------------+                       +------------------+
  | Admin SQ/CQ      | <--- one pair --->    | Admin SQ/CQ      |
  +------------------+                       +------------------+
  | I/O SQ 1 / CQ 1  | <--- per-core --->    | I/O SQ 1 / CQ 1  |
  | I/O SQ 2 / CQ 2  |                       | I/O SQ 2 / CQ 2  |
  | ...              |                       | ...              |
  | I/O SQ N / CQ N  |                       | I/O SQ N / CQ N  |
  +------------------+                       +------------------+
        |                                          |
        +---- Transport: PCIe / RDMA / TCP / FC ---+

  Each queue pair is independent. No locks between cores.
  Commands are 64B; completions are 16B; identical on every transport.

The fabrics layer adds two things and only two things to this picture. First, a small set of fabric-specific admin commands (Connect, Property Get/Set, Authentication Send/Recv) so a host can establish a controller session over a network. Second, a way to carry data payloads alongside commands, since unlike PCIe you can’t just hand the controller a host memory address and expect it to DMA into it — at least not directly.

That second part is the entire transport question. How does the data move?


RDMA, TCP, and Fibre Channel: pick your trade-off

NVMe-oF has three production transports. They share 100% of the command set and 0% of the operational profile.

NVMe/RDMA uses Remote Direct Memory Access — usually RoCEv2 over Ethernet, sometimes iWARP, sometimes native InfiniBand. The point of RDMA is that the NIC reads and writes the host’s memory directly, with no CPU involvement on either side after the initial setup. A command says “here is a memory region key and an offset; put the data there,” and the NIC does it. CPU usage is tiny; latency is the lowest of any IP-based option; throughput scales nearly linearly with cores. The catch is that RoCEv2 needs a network that does not drop packets, because RDMA reacts to loss by collapsing the queue pair rather than retransmitting. That means a Data Center Bridging (DCB) configuration with Priority Flow Control (PFC) and Explicit Congestion Notification (ECN) on every switch in the path, careful ETS bandwidth assignment, and a fabric where you actually have administrative control over every hop.

NVMe/TCP runs the NVMe queue model over a vanilla TCP connection. No special NICs, no PFC, no fabric tuning. The host kernel runs a TCP-based transport driver that takes commands and PDUs and shoves them down a socket. CPU cost is real but not catastrophic on modern hardware, especially with TLS offload and CRC offload in the NIC. Latency is higher than RDMA — typical numbers are 150–300µs round-trip versus 30–100µs for RoCE — but for the vast majority of workloads it is dramatically better than iSCSI on the same hardware. NVMe/TCP is what made NVMe-oF go from “boutique HPC protocol” to “default for new flash arrays” between roughly 2020 and 2024.

NVMe/FC runs NVMe over Fibre Channel using FC-NVMe encapsulation. If you already have a Fibre Channel SAN, this is the cheap upgrade: the same HBAs, switches, zoning, and operational practices keep working, but you get NVMe semantics on the wire and on the controller. There is no SCSI translation, multi-queue is honored end-to-end, and most modern arrays can run FC-NVMe and FCP (SCSI) over the same fabric simultaneously while you migrate workloads.

Transport Latency (typical) CPU per IOP Network requirements Where it lives in 2026
NVMe/RDMA (RoCEv2) 30–100µs Very low (offload) Lossless Ethernet, PFC + ECN, DCB tuning, RDMA-capable NIC AI training fabrics, HPC, low-latency databases, hyperconverged Ceph clusters
NVMe/RDMA (InfiniBand) 15–50µs Very low Dedicated IB fabric, subnet manager HPC, GPU clusters with NVIDIA Quantum-2/X800
NVMe/RDMA (iWARP) 50–150µs Low Standard Ethernet, no PFC needed Niche; mostly Chelsio installs
NVMe/TCP 150–300µs Moderate Any IP network, ideally 25GbE+ Default for new general-purpose enterprise SAN, cloud block storage backends
NVMe/FC 50–200µs Low (HBA offload) Existing FC SAN, zoning Enterprise shops upgrading from FCP without ripping out the fabric

The honest summary: RDMA is the fastest but the most operationally expensive; TCP is the default for everything that doesn’t have a hard latency budget; FC keeps existing investments alive. If you’re starting fresh on Ethernet with general-purpose workloads, NVMe/TCP is what you should be deploying.


Why TCP took over

NVMe/TCP did not exist when NVMe-oF was first ratified in 2016. The original spec covered RDMA and FC only. The TCP binding was added in NVMe-oF spec 1.1 in November 2018, and it was added because the people deploying NVMe-oF in real datacenters kept running into the same problem: RoCEv2 is hard.

The promise of RoCE is that you take Ethernet you already have and run RDMA on it. The reality is that Ethernet was not designed to be lossless, and RoCEv2 was not designed to tolerate loss. PFC at scale has well-documented pathological behaviors: PFC storms, head-of-line blocking, congestion spreading across the fabric because a slow downstream device pauses the entire upstream path. Microsoft’s Azure RoCE team published a paper documenting exactly these failure modes and the engineering work required to keep them under control. The Google “1RMA” paper later argued for moving away from PFC entirely and using a custom transport instead. You can make RoCE work, but it requires a network engineering team that knows DCB intimately, switches with mature buffer telemetry, and a willingness to debug correlated outages.

NVMe/TCP eliminated all of that. The Linux kernel target landed in 5.0 (January 2019), the initiator in the same release, and within two years every major storage vendor — Pure, NetApp, Dell EMC PowerStore, Hitachi, Lightbits, IBM — had shipped NVMe/TCP support. Lightbits in particular built an entire company around the idea that NVMe/TCP on commodity Ethernet would be the default replacement for traditional SAN fabrics, and the rest of the industry largely followed.

The TCP transport pays a real cost in latency and CPU. It is not free. But it runs on any Ethernet that already works for everything else, requires no fabric reconfiguration, and crosses Layer 3 boundaries cleanly. For most workloads, “30% slower than RoCE but works on the network we already have” is the correct trade-off.

   TCP PDU layout (NVMe/TCP)
   =========================

   +----------------+------+
   | Common Header  | HDGST|     8 bytes header + optional 4B digest (CRC32C)
   +----------------+------+
   | PDU-specific   |      |     Variable; depends on PDU type:
   | header         |      |       CapsuleCmd, CapsuleResp,
   |                |      |       H2CData, C2HData, H2CTermReq, etc.
   +----------------+------+
   | Data (if any)  | DDGST|     Variable + optional 4B data digest
   +----------------+------+

   Each NVMe queue pair = one TCP connection.
   Per-core queues = per-core TCP sockets = parallelism by default.

The two digests (HDGST and DDGST) are CRC32C checksums covering the header and data sections of every PDU. They are optional but on by default in most stacks. Modern NICs offload the CRC32C computation, so even with both digests enabled the CPU cost is small.


The Linux target and host: a working configuration

The Linux kernel includes both an NVMe-oF target (nvmet) and an NVMe-oF host driver. The target supports TCP, RDMA, and FC transports. The host supports the same. Configuration on the target side goes through configfs under /sys/kernel/config/nvmet/, and a small CLI called nvmetcli makes that survivable.

Here is a minimal NVMe/TCP target exporting a single SSD as namespace 1 of a subsystem:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# On the target host
modprobe nvmet
modprobe nvmet-tcp

# Define the subsystem (the unit a host connects to)
cd /sys/kernel/config/nvmet
mkdir -p subsystems/nqn.2026-06.io.lunarops:storage01
cd subsystems/nqn.2026-06.io.lunarops:storage01
echo 1 > attr_allow_any_host

# Add a namespace backed by a real block device
mkdir -p namespaces/1
cd namespaces/1
echo -n /dev/nvme0n1 > device_path
echo 1 > enable

# Define a TCP port and link the subsystem to it
cd /sys/kernel/config/nvmet
mkdir -p ports/1
cd ports/1
echo 10.0.10.5     > addr_traddr
echo tcp           > addr_trtype
echo ipv4          > addr_adrfam
echo 4420          > addr_trsvcid
ln -s ../../subsystems/nqn.2026-06.io.lunarops:storage01 subsystems/storage01

Port 4420 is the IANA-assigned default for NVMe/TCP. Port 4421 is the default for NVMe Discovery service. The NQN (NVMe Qualified Name) is the unique identifier for a subsystem — the equivalent of a SCSI IQN, and the value the host will reference when connecting.

On the host:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# Discover what's available on the target
modprobe nvme-tcp
nvme discover -t tcp -a 10.0.10.5 -s 4420

# Connect to the subsystem
nvme connect -t tcp \
  -a 10.0.10.5 -s 4420 \
  -n nqn.2026-06.io.lunarops:storage01

# Verify
nvme list
# Expected: /dev/nvme1n1  Linux  Linux  ...

# When done
nvme disconnect -n nqn.2026-06.io.lunarops:storage01

That’s it. After the nvme connect, the host sees /dev/nvme1n1 and treats it exactly like a local NVMe namespace. fio, blkdiscard, partitioning, filesystems, LVM — everything just works because the kernel does not know it’s remote.

For RoCEv2 the only thing that changes is the transport type — tcp becomes rdma, the modules become nvmet-rdma and nvme-rdma, and the underlying network must be configured for lossless behavior on the chosen traffic class. The NVMe-oF commands themselves are identical.


Namespaces, NQNs, and access control

The data model has three nested objects, and getting them straight saves a lot of grief later.

A subsystem is the unit a host connects to. It has an NQN, exposes some number of namespaces, and is reached through one or more ports. A subsystem is roughly the analog of an iSCSI target.

A namespace is a single block device inside a subsystem. It has a Namespace ID (NSID) within the subsystem and a globally unique NGUID/UUID that travels with it. A host that connects to a subsystem sees all the namespaces that have been allowed for its host NQN. Namespaces are the analog of LUNs.

A port is a network endpoint — an (IP, port, transport) tuple on the target. A subsystem can be exported through multiple ports for multipath. A port can carry multiple subsystems.

Host access control is by host NQN. The host’s NQN is normally read from /etc/nvme/hostnqn and is set once at install time. A subsystem can either allow any host (attr_allow_any_host = 1) or maintain an allowlist of permitted host NQNs:

1
2
3
4
5
6
# On the target: restrict by host NQN
cd /sys/kernel/config/nvmet/subsystems/nqn.2026-06.io.lunarops:storage01
echo 0 > attr_allow_any_host
mkdir -p ../../hosts/nqn.2026-06.io.lunarops:host-app01
ln -s ../../hosts/nqn.2026-06.io.lunarops:host-app01 \
      allowed_hosts/host-app01

For real production, you also want NVMe-oF authentication (DH-HMAC-CHAP, added in NVMe 2.0) and in-band TLS. Both are now in mainline Linux. CHAP gives you mutual authentication between host and controller; TLS encrypts the wire. They are not on by default and the configuration is fiddlier than it should be, but they exist.

NVMe-oF can also encrypt the namespace itself — the so-called Secure Channel — but in practice most deployments push encryption down to the SSD’s hardware AES engine and use NVMe-oF transport security only for authentication and integrity. That keeps the CPU cost predictable.


Multipath: ANA, not dm-multipath

The Linux NVMe stack has its own native multipath layer that is not dm-multipath. The protocol-level mechanism it implements is Asymmetric Namespace Access (ANA), defined in the NVMe base specification. ANA is the NVMe answer to ALUA for SCSI — a way for the controller to tell the host which paths are optimized, which are non-optimized, and which are inaccessible, on a per-namespace basis.

When a host connects to a subsystem through multiple ports, each path reports an ANA state for every namespace:

  • Optimized — preferred, send I/O here.
  • Non-Optimized — works, but penalized.
  • Inaccessible — path is up but the namespace is not reachable.
  • Persistent Loss — namespace lost; do not retry.
  • Change — state is changing; defer.

The kernel coalesces namespaces that share the same NSID/NGUID across paths into a single /dev/nvmeXnY block device and selects which underlying path to dispatch each I/O to according to a policy. The current policies are numa (default — pick the path closest to the CPU issuing the I/O), round-robin, and queue-depth (pick the path with the fewest outstanding I/Os, added in kernel 6.2). All policies prefer ANA-optimized paths.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Inspect ANA state on a connected host
nvme list-subsys /dev/nvme1n1

# Example output:
# nvme-subsys1 - NQN=nqn.2026-06.io.lunarops:storage01
# +- nvme1 tcp traddr=10.0.10.5 trsvcid=4420 live optimized
# +- nvme2 tcp traddr=10.0.20.5 trsvcid=4420 live non-optimized

# Change the multipath policy for a subsystem
echo round-robin > /sys/class/nvme-subsystem/nvme-subsys1/iopolicy

If you used to admin SCSI SANs, the most important thing to internalize is this: multipath -ll does not work on NVMe-oF. There is no dm-multipath device. There is a single /dev/nvmeXnY and underneath it the kernel manages the path table itself. You inspect it with nvme list-subsys, not multipathd.

The other thing to internalize: the kernel does not retry I/O on a different path indefinitely. If a controller goes inaccessible, I/O is queued for a bounded time controlled by ctrl_loss_tmo (default 600 seconds) and fast_io_fail_tmo (default off). Tune these. For HA workloads, set fast_io_fail_tmo to something like 5–15 seconds so I/O fails over instead of hanging.


Discovery: the part that used to be terrible

Discovery is how a host figures out what subsystems it can connect to. Conceptually it is simple: ask a Discovery Controller, get back a list of (transport, address, NQN) tuples, connect to the ones you care about.

In practice, for years it was awful. You had to manually configure every host with the IP of every target, run nvme discover for each, then nvme connect-all, then write a /etc/nvme/discovery.conf to make it persistent across reboots, then maintain that file across hundreds of hosts as the storage layout changed. Some shops just wrote Ansible to brute-force it.

The fix is the Centralized Discovery Controller (CDC), defined in the NVMe TP 8010 series and ratified into the base spec in 2022. A CDC is a network service — sometimes a switch-resident process, sometimes a dedicated VM, sometimes a feature of the storage array — that aggregates registrations from every storage subsystem on the fabric. Hosts find the CDC automatically via mDNS, register themselves, and get pushed a per-host view of what they’re allowed to see. When a new namespace appears, every authorized host learns about it without manual intervention.

   Centralized Discovery (TP 8010 flow)
   =====================================

   1. Storage subsystem boots, registers with CDC over a control connection
   2. Host boots, discovers CDC via mDNS (_nvme-disc._tcp)
   3. Host establishes persistent connection to CDC, registers itself
   4. CDC pushes "you may access these subsystems" log page to host
   5. Host runs connect-all against the discovered subsystem list
   6. When storage layout changes, CDC sends asynchronous event to hosts

On Linux, the user-space side of this is the nvme-stas project (stasd, stafd, stacd) which automates mDNS-based CDC discovery, persistent connection, and reconnect-on-event. It is now packaged in every major distribution. If you set up NVMe/TCP today and you are not running stasd, you are doing manual discovery and you are doing it wrong.

The CDC story is also why NVMe/TCP became operationally viable at scale. Without automated discovery, manually plumbing a thousand hosts to a hundred subsystems is intolerable. With it, the experience is roughly: “the array shows up, the host shows up, they find each other, namespaces appear.” Switch vendors — Dell SmartFabric, Cisco, Aruba — now embed CDC services directly in the switches, so the storage admin does not need to operate a separate discovery VM.


Where NVMe-oF actually beats iSCSI

iSCSI is not dead. It is the most widely supported network block protocol in existence, every hypervisor speaks it, and on small workloads the performance is fine. But the gap is real, and it has gotten wider as NVMe drives have gotten faster.

The performance differences come from three places. First, queue parallelism. iSCSI uses one TCP connection per session and serializes commands through that one queue with a small Command Sequence Number tag space. NVMe-oF uses one queue pair per CPU core, each with its own connection on TCP or its own QP on RDMA, all dispatching in parallel. On modern multi-core systems this is a 4–8x throughput difference before any optimization.

Second, command translation. iSCSI carries SCSI commands. If the underlying storage is NVMe (which on any modern array it is), every command incurs a SCSI-to-NVMe translation on the controller. NVMe-oF carries NVMe commands natively. The savings are tens of microseconds per I/O — small in absolute terms, but on a 4K random read budget of 100µs total that is a 20–30% gain.

Third, CPU cost on the host. iSCSI’s per-I/O cost is higher because of the session state machine, SCSI midlayer, and tag management. SPDK published numbers showing the SPDK iSCSI target at about 1.33M IOPS per core vs. roughly 8x higher for NVMe-oF with the same target architecture, and the Linux kernel iSCSI initiator is significantly less efficient than the SPDK target.

Property iSCSI NVMe/TCP NVMe/RDMA
Random 4K read latency (best case) 250–500µs 150–300µs 30–100µs
IOPS per CPU core (host) ~150k ~400k ~1M+
Queues per session 1 (with tagged commands) N (one per host CPU) N (one per host CPU)
Wire encoding SCSI in iSCSI PDU NVMe in TCP PDU NVMe in RDMA send/RDMA write
Network requirements Any IP Any IP Lossless Ethernet or InfiniBand
Discovery automation iSNS (rare) or static mDNS + CDC + nvme-stas mDNS + CDC + nvme-stas
OS support Universal Linux 5.0+, Windows Server 2025, VMware 7.0+, FreeBSD 14+ Linux, Windows, VMware, ESXi (limited iWARP)
Operational maturity 20 years 5–6 years and rapidly maturing Niche outside HPC and arrays

The honest counter: iSCSI is fine. If you have a small homelab, two or three TrueNAS shares, and a VMware host that needs block storage, iSCSI is supported by everything, debuggable with tcpdump, and the performance is more than enough. The case for NVMe-oF is real but it’s a case about the high end — where you are filling 25/100GbE links, where the workload is latency-sensitive, where you have enough hosts that discovery automation matters, or where you are buying a new array and the vendor’s recommendation is NVMe/TCP anyway.


Where it loses to local disk

The other honest comparison is against direct-attached storage. A modern PCIe Gen5 NVMe drive can do 14 GB/s sequential, 2.5M random read IOPS, and 5–10µs read latency. No fabric — not even InfiniBand — gets that latency. The minimum round-trip on the best-case RDMA fabric is around 15–20µs, and even that is best-case; with a switch in the path, normal traffic, and any queueing, you are at 30–50µs minimum. NVMe/TCP doubles that. NVMe-oF on a good network is “near” local NVMe; it is not equal to it.

The other thing local disk gives you that fabrics cannot is decoupled failure domains. A network outage takes out every host’s storage simultaneously. A switch firmware bug can do the same. A local SSD failing kills one host. The same trade-off that has applied to network storage for forty years still applies — NVMe-oF made it cheaper to centralize storage but did not change the failure correlation math.

So the use cases where NVMe-oF wins decisively are the use cases where centralized storage was already the right answer:

  • Hyperconverged virtualization — VMs need to migrate across hosts, so the disk needs to be reachable from multiple hosts.
  • Databases with shared storage HA — Oracle RAC, SQL Server FCI, anything that depends on a single block device visible to multiple nodes.
  • GPU training fabrics — Where dataset throughput is the bottleneck and the storage layer is built from JBOFs (Just a Bunch Of Flash) shared by many GPU nodes.
  • Cloud block storage backends — AWS EBS io2 Block Express, Azure Ultra Disk, GCP Hyperdisk; all are NVMe-oF under the hood, although the cloud providers don’t expose it directly.

For a workload that fits on local disk, local disk is still faster and simpler. NVMe-oF wins when the storage has to be elsewhere anyway.


A homelab-scale deployment

If you want to actually run NVMe-oF in a homelab to learn it, the smallest useful setup is two Linux machines with a 10GbE link between them. Use one as the target (export an SSD or a tmpfs-backed file), the other as the host. NVMe/TCP needs no special hardware. The kernel modules are in every major distro.

For a slightly more interesting setup, two targets sharing a namespace via ANA gives you working multipath on cheap gear. The detail to get right is namespace identity — both targets must export the same NGUID/UUID so the host coalesces them. With nvmet, write the same UUID into both namespaces:

1
2
3
# On both targets
echo "12345678-1234-1234-1234-123456789abc" > \
  /sys/kernel/config/nvmet/subsystems/.../namespaces/1/device_uuid

Now nvme connect to both targets in turn, and the host sees a single /dev/nvme1n1 with two underlying paths. Pull one network cable and watch the I/O continue without dropping a beat.

   Homelab NVMe/TCP multipath topology
   ====================================

         +-----------------------+
         |  Host: app-server     |
         |  /dev/nvme1n1 (ANA)   |
         +----+--------------+---+
              |              |
        10GbE |              | 10GbE
              |              |
   +----------+--+      +----+-----------+
   | nvmet      |      | nvmet           |
   | target-A   |      | target-B        |
   | (optimized)|      | (non-optimized) |
   +------------+      +-----------------+

   Same NGUID on namespace 1 on both targets.
   Native NVMe multipath coalesces them.
   ANA tells the host which is preferred.

For RoCE in a homelab, ConnectX-3 cards are cheap on the secondary market, but the switch question is harder — most cheap 10/25GbE switches do not implement PFC properly, and without PFC, RoCEv2 will degrade unpredictably under load. If you do not have a switch that supports PFC and ECN correctly, run RoCE only over a direct cable between two cards. Otherwise stick with NVMe/TCP.


The performance math you actually need

For sizing, the equation is roughly:

total_iops = min(
    network_bandwidth / io_size,
    target_storage_iops,
    host_cpu_cycles_per_second / cycles_per_io,
    queues * queue_depth * (1 / round_trip_latency)
)

The fourth term — the queue/latency product — is what NVMe-oF improves dramatically. With iSCSI’s single queue of depth 32 and a 500µs round trip, you cap at 64k IOPS per session. With NVMe-oF on 8 cores each running a queue of depth 128 at 150µs round trip, you get roughly 6.8M IOPS theoretical — three orders of magnitude headroom before the network or storage becomes the bottleneck.

In practice the limit becomes the actual SSD or the network bandwidth long before the queue product caps out. A 25GbE link at 4K I/O size is 25 × 10^9 / 8 / 4096 ≈ 760k IOPS at line rate. A single high-end NVMe drive can do 2.5M random read IOPS. So the link is the bottleneck, not the queues — which is exactly the situation iSCSI never gave you.

For mixed read/write workloads with mid-size I/O (16K–64K), a single 25GbE NVMe/TCP link can comfortably saturate one or two SSDs. For sequential streaming, throughput is bandwidth-bounded and a single connection will hit ~22 Gbps on a 25GbE link with reasonable tuning. For 4K random reads, latency dominates and you cap at the link’s IOPS ceiling. None of this is hand-wavy — these numbers reproduce on any properly tuned target.


Verdict

NVMe-over-Fabrics is the first network block protocol that does not feel like a network protocol — it preserves the multi-queue, low-overhead nature of local NVMe and adds tens of microseconds of latency for the privilege of running the storage somewhere else. NVMe/TCP is the right default for greenfield deployments because it requires no fabric reconfiguration, runs on any IP network, and is supported by every modern storage stack. Use NVMe/RDMA when you have a controlled fabric and a workload that cares about every microsecond — AI training, low-latency databases, hyperconverged clusters with internal storage networks. Use NVMe/FC only if you already have a Fibre Channel SAN and want to keep it.

Plain iSCSI is not dead, and for small homelab use cases it is still simpler and universally supported. But if you are sizing new storage in 2026, NVMe/TCP is what your arrays already speak, what your hypervisors increasingly prefer, and what the discovery story is finally pleasant on. The biggest operational surprise for SCSI veterans is that dm-multipath does not apply — Linux NVMe multipath is its own subsystem driven by ANA, configured via sysfs, and inspected with nvme list-subsys. Internalize that and most of the day-2 friction disappears.

For deeper reading on the ingredients that make NVMe-oF work, see the RDMA and InfiniBand post for the underlying memory-semantics model that NVMe/RDMA inherits, Ceph as software-defined storage for how distributed block storage uses these transports at scale, the Flash Translation Layer for what’s happening inside the SSDs you are now reaching over the wire, Thunderbolt and USB4 tunneling for the analogous “encapsulate a bus protocol over a different transport” pattern in the consumer world, and CXL memory pooling for where the disaggregation story goes next.

Sources

Comments