Training Cluster Networking: NVLink, Rails, and the Bandwidth Budget That Bounds Model Scale

June 21, 2026 15 min read AI & Machine Learning

gpudistributed-traininginfinibandnvlinkrocerail-optimizedmachine-learning

Contents

The uncomfortable truth of large-scale training is that you do not buy FLOPs, you buy a network that happens to have GPUs bolted to it. A modern accelerator can sustain on the order of a petaflop of dense math, but a training step is a sequence of matrix multiplies punctuated by collective communication — gradients summed, activations exchanged, experts routed — and the moment those collectives cannot hide behind compute, every GPU in the job sits idle waiting for the slowest link. At eight GPUs this is a tuning detail. At eight thousand it is the entire engineering problem. This post is about the fabric that decides whether your cluster scales: the two distinct networks every training system actually runs, why tensor parallelism must live inside an NVLink domain while data parallelism can cross a slower fabric, how rail-optimized topology and in-network reduction buy back bandwidth, and the arithmetic that tells you, before you sign the purchase order, whether the network will keep your GPUs fed. It is the architectural companion to the mechanics in RDMA and InfiniBand from the ground up and the collective-software internals in the NCCL deep dive.

Two Networks, Not One

The single most important fact about a GPU cluster is that it has two interconnects with bandwidths an order of magnitude apart, and the gap between them dictates everything else. The first is the scale-up network: NVLink plus NVSwitch, which fuses a handful of GPUs (8 in a Hopper DGX node, 72 in a Blackwell NVL72 rack) into a single coherent domain where any GPU can read any other GPU’s HBM at terabytes per second. The second is the scale-out network: InfiniBand or RoCE over Ethernet, which stitches those domains into a cluster of thousands across a datacenter at hundreds of gigabytes per second per node.

That is roughly a 10x bandwidth cliff at the edge of the NVLink domain, and it is not a flaw to be engineered away — it is the defining constraint you design the whole training job around. Collectives that demand the most bandwidth get placed inside the fast domain; collectives that tolerate less get pushed across the slow fabric. Get that mapping wrong and you can leave half your cluster idle while the interconnect saturates.

Tier	Technology (2026)	Per-GPU / per-port BW	Latency	Scope
Scale-up	NVLink 4 (Hopper)	~900 GB/s aggregate	~hundreds of ns	8-GPU node
Scale-up	NVLink 5 + NVSwitch (Blackwell)	~1.8 TB/s aggregate	~hundreds of ns	up to 72-GPU rack
Scale-out	InfiniBand NDR	400 Gb/s (~50 GB/s)	~1-2 us	whole cluster
Scale-out	InfiniBand XDR	800 Gb/s (~100 GB/s)	~1-2 us	whole cluster
Scale-out	RoCEv2 / Spectrum-X	400-800 Gb/s	~2-3 us	whole cluster
Fallback	PCIe Gen5 x16	~64 GB/s	~us	within host

The numbers move every generation, but the ratio is durable: intra-domain bandwidth is roughly 20-40x a single scale-out port, and even with eight NICs per node the aggregate node injection bandwidth is still several times below NVLink. Designing as if the cluster were one flat network is the most common and most expensive mistake.

Inside the Node: NVLink and NVSwitch

NVLink is a point-to-point serial link, but the thing that makes a node useful is NVSwitch, the crossbar ASIC that turns a pile of NVLink-capable GPUs into a non-blocking all-to-all. In a Hopper-class 8-GPU server, four NVSwitch chips connect all eight GPUs so that every pair gets the full ~900 GB/s of NVLink 4 bandwidth simultaneously, with no oversubscription. This is what lets a collective like all-reduce run at “bus bandwidth” inside the node: every GPU pushes and pulls at line rate at once.

The reason this matters for model design is tensor parallelism. When you split a single matrix multiply across GPUs — sharding the weight matrix and having each GPU compute a slice — every layer’s forward and backward pass ends in an all-reduce of the partial activations. That is communication on the critical path of every single layer, with no opportunity to overlap it behind unrelated compute. Tensor parallelism is only viable where the all-reduce is nearly free, which means it must stay inside the NVLink domain. Stretch a tensor-parallel group across the scale-out fabric and the per-layer all-reduce latency dominates; throughput collapses. This single rule — TP lives in the NVLink domain, period — explains most of the topology decisions that follow.

You can see the domain structure directly:

$ nvidia-smi topo -m
        GPU0   GPU1   GPU2   GPU3   NIC0   NIC1
GPU0     X     NV18   NV18   NV18   PXB    SYS
GPU1    NV18    X     NV18   NV18   SYS    PXB
GPU2    NV18   NV18    X     NV18   SYS    SYS
GPU3    NV18   NV18   NV18    X     SYS    SYS
...
Legend: NV# = NVLink (# links)  PXB = PCIe switch  SYS = across host bridge

NV18 means eighteen NVLink connections between that GPU pair through the switch fabric — the fast path. SYS means the only route is across the CPU host bridge, which is the path you never want a hot collective to take.

The NVL72 Inflection: When the Rack Is the GPU

For most of the last decade the NVLink domain was a single server: eight GPUs, then you hit the wall and everything beyond was InfiniBand. The Blackwell generation changes the unit. GB200 NVL72 packages 72 Blackwell GPUs and 36 Grace CPUs into one liquid-cooled rack wired as a single NVLink domain through a spine of NVSwitch trays, delivering on the order of 130 TB/s of aggregate NVLink bandwidth inside the rack. The rack, not the server, becomes the atom of training.

This is not a marketing distinction. A 72-GPU coherent domain means your tensor-parallel and a chunk of your other high-bandwidth parallelism can run entirely at NVLink speed before you ever touch the slow fabric. A model that needed TP=8 inside a node plus pipeline parallelism across nodes can now fit a far larger fast-domain footprint, shrinking the fraction of communication that must cross InfiniBand. The practical consequence: the same scale-out fabric now connects fewer, larger domains, so the cluster’s external collective volume per FLOP drops. The flip side is blast radius and cost — a single NVL72 is a $3M-class unit, liquid cooling is mandatory, and a fault in the NVLink spine can take 72 GPUs out of a synchronous job at once.

Scale-Out: InfiniBand, RoCE, and the Rail-Optimized Pod

Beyond the NVLink domain, GPUs talk over RDMA — either InfiniBand or RoCEv2 on lossless Ethernet (NVIDIA’s Spectrum-X is the productized RoCE-for-AI variant). The defining design pattern of a modern AI pod is one NIC per GPU, each at 400 or 800 Gb/s, and a rail-optimized wiring of those NICs into the leaf layer.

Rail-optimization is the idea that pays the rent. Number the GPUs in each node 0 through 7. Every GPU-0 across every node in a pod connects to the same leaf switch (“rail 0”); every GPU-1 to rail 1; and so on. The payoff: when data-parallel ranks that occupy the same position across nodes perform an all-reduce, their traffic stays on a single rail and crosses exactly one switch hop, never touching the spine. Only traffic that must move between rails climbs to the spine. Because most well-placed collectives are same-rail, this slashes the spine bandwidth you have to buy.

        RAIL-OPTIMIZED POD (8 GPUs/node, 8 rails)

                    +-------------------+
                    |   SPINE SWITCHES  |   (cross-rail only)
                    +---+---+---+---+---+
                        |   |   |   |
        +-------+-------+   |   |   +-------+-------+
        |       |       |   |   |       |       |
     [Leaf0] [Leaf1] [Leaf2]... ...[Leaf6] [Leaf7]   <- one leaf per rail
        |       |       |               |       |
   node A: G0----G1------G2  ...  ...   G6 ---- G7
   node B: G0----G1------G2  ...  ...   G6 ---- G7
   node C: G0----G1------G2  ...  ...   G6 ---- G7
        ^
   NIC per GPU; GPU#k on every node homes to Leaf#k (rail k).
   Same-rail all-reduce = 1 hop. Cross-rail = up to spine.

The leaf-spine itself is a fat-tree (folded Clos). A fully non-blocking fat-tree provisions equal bandwidth up to the spine as down to the hosts — 1:1 — so any node can talk to any other at line rate. That is expensive at scale, so some clusters accept oversubscription (say 2:1 or 4:1) at the spine, betting that rail-optimization and good rank placement keep most traffic local. The bet is sound for data-parallel-heavy jobs and dangerous for all-to-all-heavy ones (see expert parallelism below).

Mapping Parallelism Onto Cables

Every distributed-training framework exposes the same handful of parallelism axes, and each has a communication signature that decides where it belongs in the hierarchy. Placing them is the core of cluster design.

   PLACEMENT (typical 4D parallelism on rail-optimized pods)

   Tensor Parallel (TP)  -> inside NVLink domain   [highest BW, per-layer all-reduce]
   Expert Parallel (EP)  -> NVLink domain / 1 rail  [all-to-all, bisection-hungry]
   Pipeline Parallel(PP) -> across a few nodes      [point-to-point, cheap, latency-tolerant]
   Data Parallel  (DP)   -> across the rest         [one gradient all-reduce/step, overlappable]

Tensor parallelism, as established, must stay in the fast domain. Expert parallelism — the routing of tokens to a subset of feed-forward experts in a mixture-of-experts model — is an all-to-all exchange, the single most bisection-bandwidth-hungry pattern there is, because in the worst case every GPU sends to every other GPU. EP wants to stay inside the NVLink domain or, failing that, on a single rail with full bisection. Pipeline parallelism sends only layer-boundary activations between consecutive stages — point-to-point, modest volume, tolerant of the slower fabric — so it crosses nodes happily. Data parallelism replicates the model and does one gradient all-reduce per step; it is bandwidth-heavy but, crucially, overlappable with the backward pass, which is the property that makes thousand-GPU data-parallel scaling work at all. The patterns themselves are covered in FSDP and DDP distributed training; the point here is that the network topology is what makes a given mapping fast or fatal.

The Bandwidth Budget

You can predict whether a job will scale before running it, with arithmetic. Take ring all-reduce, NCCL’s workhorse for gradient synchronization. To all-reduce a buffer of S bytes across N GPUs, each GPU sends and receives 2 * (N-1)/N * S bytes over the wire — for large N that approaches 2S. The time is that volume divided by the per-GPU bus bandwidth B (the bandwidth the ring can actually sustain end to end, always below the raw link rate):

   t_allreduce  ~=  (2 * (N-1)/N * S) / B   ~=  2S / B   for large N

Now the scaling test. Data-parallel gradient all-reduce moves S = model_param_bytes per step. Suppose a 70B-parameter model in bf16: roughly 140 GB of gradients. Over a fabric where the sustained inter-node bus bandwidth is ~45 GB/s (a 400G NDR rail after protocol overhead), a naive full all-reduce is 2 * 140 / 45 ~= 6.2 s. If your step’s compute is only 2 seconds, communication dominates 3-to-1 and you are network-bound — the GPUs idle two-thirds of the time. The fixes are exactly the levers real systems pull: shard the all-reduce so each rank handles a slice (FSDP reduce-scatter plus all-gather moves the same total but overlaps it in pieces), overlap communication with backward compute so the 6 seconds hides behind the 2 seconds of the next microbatch, and reduce the volume with in-network reduction. The rule of thumb that falls out: size the fabric so that, after overlap, collective time is comfortably under compute time per step. If it is not, adding GPUs makes the job slower, because more ranks means more all-reduce volume against a fixed link rate.

In-Network Reduction: SHARP and Why the Switch Does Math

The most elegant bandwidth win is to stop moving data you are only going to add together. NVIDIA’s SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) pushes the reduction operation into the InfiniBand switch ASIC (Quantum-2 for NDR, Quantum-X800 for XDR). Instead of every GPU streaming its gradient buffer up to a root and back down, the switches build an aggregation tree, sum partial results as they pass through, and multicast the final result back. The all-reduce volume on the host links roughly halves, and the reduction latency drops because it happens at wire speed in hardware rather than in GPU kernels.

SHARP is not free to operate. It requires the aggregation tree to be provisioned by the subnet manager, it consumes switch resources, and it helps most for the large, regular all-reduces of data-parallel training — not the irregular all-to-all of expert routing. But for a data-parallel-dominated job at scale it is one of the few ways to beat the 2S/B wall without buying more cables. RoCE-based fabrics chase the same idea with their own in-network compute and careful congestion control, since lossless Ethernet for AI lives or dies on PFC and ECN tuning to avoid the incast collapse that RDMA cannot tolerate.

Inspecting and Tuning a Real Fabric

When a training run underperforms, the fabric is the first suspect, and the diagnosis is concrete. Confirm the scale-out links are up and at rate:

$ ibstat
CA 'mlx5_0'
    Port 1:
        State:           Active
        Physical state:  LinkUp
        Rate:            400          # NDR; 800 = XDR. Anything lower is a fault.
        Base lid:        0x12

$ nvidia-smi nvlink --status -i 0
GPU 0: link 0: 26.562 GB/s  ... link 17: 26.562 GB/s   # all 18 links present

A missing NVLink, a port that negotiated down to 200G, or a NIC that landed on the wrong NUMA node relative to its GPU all show up here and all silently halve throughput. Then verify the software actually rides the topology you paid for. NCCL reads the system topology and picks transports; you confirm and constrain it with environment variables:

export NCCL_DEBUG=INFO            # prints the chosen rings/trees and transports
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3   # bind to the GPU-local NICs
export NCCL_NET_GDR_LEVEL=PIX     # require GPUDirect RDMA where PCIe allows
export NCCL_TOPO_FILE=/etc/nccl-topo.xml          # hand NCCL the real fabric map

Finally, measure the ceiling with the standard benchmark before blaming your model:

$ all_reduce_perf -b 8 -e 8G -f 2 -g 8
#       size      time   busbw
#      bytes        us    GB/s
   1073741824     ...    ~370    # in-node, NVLink: should approach NVLink bus BW
   8589934592     ...     ~44    # multi-node, IB: should approach per-rail line rate

If busbw is far below the line-rate expectation for the relevant tier, the problem is the fabric (cabling, routing, congestion, GDR not engaged), not your training loop. The full set of knobs lives in the NCCL deep dive; the discipline is to compare measured busbw against the tier’s theoretical bus bandwidth and chase the gap.

Where It Breaks at Scale

Synchronous data-parallel training has a brutal property: it runs at the speed of the slowest participant, every step. At eight thousand GPUs, the probability that something is degraded on any given step approaches one. A single NIC that flaps, a cable that develops bit errors and forces retransmits, an optical transceiver running hot and lowering its effective rate — any of these becomes a straggler that stalls the entire job, and the symptom is maddeningly diffuse: average utilization sags a few percent with no obvious culprit. Large operators run continuous fabric validation precisely because finding the one bad link in a haystack of a hundred thousand is the daily reality.

Two more failure modes deserve naming. ECMP hash collisions: classic equal-cost multipath routing hashes flows to spine links, and with the small number of enormous, long-lived flows that training produces, two elephant flows can hash to the same link and congest it while a parallel link sits idle — which is why AI fabrics lean on adaptive routing and rail-optimization rather than vanilla ECMP. Incast: in an all-to-all or an all-reduce’s gather phase, many senders target one receiver simultaneously, overrunning buffers; InfiniBand’s credit-based flow control handles this natively, while RoCE must lean on PFC and ECN tuned carefully enough to avoid both drops and head-of-line blocking. The bandwidth-versus-power ceiling looming over all of this is why the industry is moving toward co-packaged optics and silicon photonics: pluggable transceivers at 800G and beyond burn too much power and fail too often at the scale these clusters now demand, and the same memory-bandwidth pressure shows up one level down in HBM and the memory wall.

Verdict

Treat the network as the primary design surface of a training cluster, not an afterthought you bolt on once the GPUs arrive. The decisions that determine whether your accelerators run at 90% utilization or 40% are made in the topology: keep tensor and expert parallelism inside the NVLink domain where bandwidth is effectively free, let pipeline parallelism span nodes because it is cheap and latency-tolerant, and reserve the scale-out fabric for the overlappable gradient all-reduce of data parallelism. Build the pod rail-optimized so most collectives cross one switch hop, provision the fat-tree as close to non-blocking as the budget allows if your jobs are all-to-all-heavy, and turn on in-network reduction where the collective is regular enough to benefit. Above all, do the bandwidth-budget arithmetic before you scale: if 2S/B does not fit under your per-step compute time after overlap, more GPUs will make the job slower, and no amount of kernel optimization will save you. The GB200 NVL72 generation shifts the calculus by making the rack a 72-GPU fast domain, but it does not repeal the law — it just moves the cliff. The cluster that scales is the one whose owners understood, before purchase, that they were buying a network.