Expert-Parallelism

• Jun 21, 2026 Mixture of Experts, Honestly: Why Every Frontier Model Went Sparse and What It Actually Costs

  The honest accounting on Mixture of Experts: why every frontier model in 2026 is sparse, what the "X total / Y active" parameter math actually buys you, the routing problems the marketing skips, and the all-to-all communication tax that decides whether a MoE model ships or stalls.

• May 22, 2026 Mixture of Experts Internals: Routing, Expert Parallelism, and Load Balancing

  MoE is now the default architecture for top-tier open models. This is how it actually works: router design, top-k token dispatch, auxiliary loss for load balancing, expert parallelism across GPUs, and what it means for inference serving — with DeepSeek-V3 and Mixtral as concrete case studies.