A map of the five ways to split a training job across GPUs — data, tensor, pipeline, expert, and sequence parallelism — when each one pays off, how they compose into 3D and 4D parallelism, and the communication-versus-memory math that decides the whole thing.
Moe
-
Distributed Training Parallelism -
Mixture of Experts, Honestly: Why Every Frontier Model Went Sparse and What It Actually Costs The honest accounting on Mixture of Experts: why every frontier model in 2026 is sparse, what the "X total / Y active" parameter math actually buys you, the routing problems the marketing skips, and the all-to-all communication tax that decides whether a MoE model ships or stalls.
-
Mixture of Experts Internals: Routing, Expert Parallelism, and Load Balancing MoE is now the default architecture for top-tier open models. This is how it actually works: router design, top-k token dispatch, auxiliary loss for load balancing, expert parallelism across GPUs, and what it means for inference serving — with DeepSeek-V3 and Mixtral as concrete case studies.
-
Llama 4 and Gemma 4: The 2026 Self-Hosted Model Landscape Meta shipped Llama 4 Scout and Maverick and Google shipped four Gemma 4 variants under Apache 2.0 — all within a few weeks of each other. This is a practical walkthrough of what each model is, what hardware you actually need, how to deploy them with Ollama, vLLM, and llama.cpp, and which one to pick for which job.