The distributed-training patterns that actually scale in PyTorch — DDP versus FSDP, how each shards parameters, gradients, and optimizer state, and the collective-communication and precision knobs that decide whether more GPUs help.
Pytorch
-
FSDP and DDP: Distributed Training Patterns That Actually Scale -
NCCL Deep Dive: Multi-GPU Collectives, Ring vs Tree, and Debugging Distributed Training How NVIDIA's NCCL moves gradients across multi-GPU and multi-node training — ring versus tree collectives, NVLink and InfiniBand topology awareness, and how to debug the silent hangs and bandwidth cliffs that take training down.
-
Fine-Tuning LLMs on Your Own Hardware LoRA and QLoRA explained, unsloth for efficient fine-tuning, dataset preparation, evaluation, merging adapters, and when fine-tuning actually beats RAG — a practical guide to training language models on consumer and prosumer hardware.