Why the network, not the GPU, decides how big a model you can train: the NVLink and NVSwitch scale-up domain versus the InfiniBand and RoCE scale-out fabric, rail-optimized topology, in-network reduction, and the bandwidth-budget math that maps tensor, pipeline, data, and expert parallelism onto real cables.
Infiniband
-
Training Cluster Networking: NVLink, Rails, and the Bandwidth Budget That Bounds Model Scale -
NCCL Deep Dive: Multi-GPU Collectives, Ring vs Tree, and Debugging Distributed Training How NVIDIA's NCCL moves gradients across multi-GPU and multi-node training — ring versus tree collectives, NVLink and InfiniBand topology awareness, and how to debug the silent hangs and bandwidth cliffs that take training down.
-
Parallel Filesystems Compared: Lustre vs BeeGFS vs GPFS vs Weka When NFS stops scaling you need a parallel filesystem — a comparison of Lustre, BeeGFS, GPFS, and Weka covering how each distributes data and metadata, their performance profiles, and the operational burden each one brings.
-
RDMA and InfiniBand From the Ground Up: Why It's Fast and How to Diagnose It Why RDMA and InfiniBand deliver 400 Gbps at single-microsecond latency with the CPU idle — how kernel-bypass and the verbs model work, how RoCE differs, and how to diagnose a fabric that isn't hitting its datasheet.