How NVIDIA's NCCL moves gradients across multi-GPU and multi-node training — ring versus tree collectives, NVLink and InfiniBand topology awareness, and how to debug the silent hangs and bandwidth cliffs that take training down.
Nccl
-
NCCL Deep Dive: Multi-GPU Collectives, Ring vs Tree, and Debugging Distributed Training