Fsdp

• Jun 21, 2026 Distributed Training Parallelism

  A map of the five ways to split a training job across GPUs — data, tensor, pipeline, expert, and sequence parallelism — when each one pays off, how they compose into 3D and 4D parallelism, and the communication-versus-memory math that decides the whole thing.

• Apr 20, 2026 FSDP and DDP: Distributed Training Patterns That Actually Scale

  The distributed-training patterns that actually scale in PyTorch — DDP versus FSDP, how each shards parameters, gradients, and optimizer state, and the collective-communication and precision knobs that decide whether more GPUs help.