Mixture of Experts

Instead of running every parameter for every token, MoE routes each token to a small group of specialist sub-networks called experts. DeepSeek V3 uses 671B total parameters but only activates 37B per token. You get frontier quality at a fraction of the compute cost.

In a dense transformer, every parameter fires for every token. In a Mixture-of-Experts transformer, the feedforward layer is replaced by a bank of experts and a router. For each token the router picks the top-k experts (usually 2 out of 64 or more) and only those experts compute. The trade-offs: load balancing loss to keep experts busy, routing instability during training, and higher total VRAM at serving time because all experts must be resident even if only a few are used per token. See the related terms and live BenchGecko data for current examples.

MoE replaces the FFN of each transformer block with N expert MLPs plus a gating network g(x) that produces a sparse top-k distribution. The forward pass becomes sum over i in top_k(g(x)) of g(x)_i * E_i(x). Auxiliary losses balance expert load (load_balancing_loss) and prevent gate dropout. Frontier MoE models push expert count to 256+ (DeepSeek V3 uses 256 routed experts + 1 shared expert). Sparsity ratio (active/total params) typically lands between 5% and 15%. Training MoE requires expert parallelism on top of tensor parallelism, and serving benefits from fused kernels like GroupedGEMM. Quality recovery relative to a dense model of the same active size is the empirical win: DeepSeek V3 at 37B active matches Llama 3.1 405B dense on most benchmarks.