A map of the five ways to split a training job across GPUs — data, tensor, pipeline, expert, and sequence parallelism — when each one pays off, how they compose into 3D and 4D parallelism, and the communication-versus-memory math that decides the whole thing.
Distributed-Training
-
Distributed Training Parallelism -
Training Cluster Networking: NVLink, Rails, and the Bandwidth Budget That Bounds Model Scale Why the network, not the GPU, decides how big a model you can train: the NVLink and NVSwitch scale-up domain versus the InfiniBand and RoCE scale-out fabric, rail-optimized topology, in-network reduction, and the bandwidth-budget math that maps tensor, pipeline, data, and expert parallelism onto real cables.
-
FSDP and DDP: Distributed Training Patterns That Actually Scale The distributed-training patterns that actually scale in PyTorch — DDP versus FSDP, how each shards parameters, gradients, and optimizer state, and the collective-communication and precision knobs that decide whether more GPUs help.
-
NCCL Deep Dive: Multi-GPU Collectives, Ring vs Tree, and Debugging Distributed Training How NVIDIA's NCCL moves gradients across multi-GPU and multi-node training — ring versus tree collectives, NVLink and InfiniBand topology awareness, and how to debug the silent hangs and bandwidth cliffs that take training down.