Why the network, not the GPU, decides how big a model you can train: the NVLink and NVSwitch scale-up domain versus the InfiniBand and RoCE scale-out fabric, rail-optimized topology, in-network reduction, and the bandwidth-budget math that maps tensor, pipeline, data, and expert parallelism onto real cables.
Nvlink
-
Training Cluster Networking: NVLink, Rails, and the Bandwidth Budget That Bounds Model Scale -
NCCL Deep Dive: Multi-GPU Collectives, Ring vs Tree, and Debugging Distributed Training How NVIDIA's NCCL moves gradients across multi-GPU and multi-node training — ring versus tree collectives, NVLink and InfiniBand topology awareness, and how to debug the silent hangs and bandwidth cliffs that take training down.