Mixture of Experts (MoE)✎

Mixture of Experts (MoE) is a machine learning architecture in which a population of independent sub-networks — the experts — each process the same input, while a learned gating function (also called a weighting or routing function) determines how their outputs are combined. The canonical formulation is a weighted sum f(x) = Σᵢ w(x)ᵢ fᵢ(x), where the gate w produces non-negative weights over the expert outputs. Both the experts and the gate are trained jointly by minimizing a loss, typically via gradient descent. MoE is a form of ensemble learning — historically also called "committee machines" — and its defining feature is that different experts can specialize: the gate learns to steer each input toward the expert(s) best suited to handle it, and the experts diverge through a positive-feedback dynamic in which better-performing experts receive stronger gradient signal and increasingly dominate their region of the input space.

The architecture spans several decades. Classical MoE (Jacobs, Jordan, Nowlan, and Hinton, 1991) and hierarchical variants (Jordan and Jacobs, 1994) predate deep learning; the deep-learning era repurposed MoE primarily as a mechanism for conditional computation — activating only a fraction of a model's parameters per token — making it practical to train models with trillions of parameters while keeping inference cost manageable. This application is now central to the largest Transformer architecture-based LLMs.

Classical MoE

The adaptive mixtures of local experts formulation uses a Gaussian mixture model: each expert predicts a Gaussian distribution over outputs, and the gate is a linear-softmax function. Training by maximum likelihood has a clean Bayesian interpretation — the posterior probability of each expert given an input–output pair acts as the effective learning rate for that expert, so experts consulted in hindsight become more specialized toward the examples they handled well, while experts that were outcompeted drift toward other regions. The combined effect is emergent specialization: experts partition the input space into local territories with little overlap.

A hierarchical extension stacks multiple levels of gating in a tree, where each node is itself a gate over the next level. This is analogous to a differentiable decision tree and enables finer-grained routing. A "hard MoE" variant, rather than computing a weighted sum of all experts, selects only the top-ranked expert — f(x) = f_{argmax w(x)} — accelerating both training and inference at the cost of non-differentiability of the selection step.

Sparse MoE in deep learning

The key motivation for MoE in deep networks is reducing per-token compute. The earliest deep MoE application (Eigen, Ranzato, and Sutskever, 2013) used a different gating network at each layer, but with a dense (non-sparse) gate — so all expert outputs were still needed. The decisive step was the sparsely-gated MoE layer (Shazeer et al., Google Brain, 2017), which achieves sparsity by retaining only the top-k experts' outputs and setting the rest to −∞ before softmax, effectively zeroing their contributions. A small amount of noise is added before the top-k selection to help load balancing. With k = 1 or k = 2, this yields strong compute savings even when the total parameter count is vastly larger than any dense alternative. The k = 1 case became the Switch Transformer (Fedus, Zoph, and Shazeer, 2022), which demonstrated scaling to trillion-parameter models with simple and efficient sparsity.

In Transformer architecture-based models, MoE layers replace the feedforward sublayer in each Transformer block — the part that grows most expensive at scale (in PaLM-540B, feedforward layers account for 90% of parameters). A trained dense Transformer can also be converted to an MoE via sparse upcycling: duplicating its feedforward layers with randomly initialized gating, then continuing training.

Routing and load balancing

Routing — deciding which tokens go to which experts — is the central engineering challenge of sparse MoE. Poor routing leads to two failure modes: some experts are perpetually overloaded while others sit idle, and some tokens may be dropped entirely. Three routing paradigms have been studied: token-choice routing (tokens pick experts, the original sparsely-gated approach), expert-choice routing (experts select the tokens they process, guaranteeing perfect load balance but allowing some tokens to be unprocessed), and global assignment (a linear program or other solver matches all tokens to experts jointly to satisfy capacity constraints).

Load imbalance is a persistent problem with vanilla MoE: through positive feedback, popular experts receive more gradient signal and become more capable, further increasing their popularity. The Switch Transformer introduced an auxiliary loss that penalizes imbalance within a batch by summing the product of each expert's token fraction and its average routing weight — minimized when all experts receive equal traffic. DeepSeek later proposed an "auxiliary-loss-free" strategy that instead adjusts a per-expert bias term: neglected experts get a higher bias so they are more likely to be selected at the top-k step, but the bias does not affect how expert outputs are weighted in the final sum.

A related hyperparameter is the capacity factor c: each expert is allowed to process at most c·(T/n) tokens in a batch of T tokens across n experts. This enforces a hard cap on overloading; values around 1.25–2.0 have been found to work well in practice (ST-MoE report, Zoph et al., 2022). Tokens routed to an at-capacity expert are dropped, but because Transformers use residual connections, dropped tokens pass through the feedforward layer unchanged rather than disappearing.

Notable implementations

Several major LLM families use sparse MoE. Google's GShard uses top-2 routing with a probabilistic second-expert selection. GLaM (Du et al., 2021) reached 1.2 trillion parameters with 64 experts per layer and top-2 routing, demonstrating competitive quality at lower inference cost than dense models. Switch Transformer uses top-1 routing throughout. Meta's NLLB-200 translation model uses a two-level hierarchical MoE: the first gate chooses between a shared feedforward layer and the expert pool; a second gate then picks the top-2 experts.

DeepSeek's architecture (DeepSeekMoE and subsequent V2/V3 releases) introduced a distinction between shared experts — always active and responsible for core capacities like general grammar — and routed experts that handle peripheral, task-specific patterns. This prevents the standard load-balancing pressure from forcing routed experts to redundantly replicate core capabilities.

Mixtral 8x7B (Mistral AI, December 2023, Apache 2.0 license) is a notable open-weight MoE: 46.7 billion total parameters, 8 experts per layer, sparsity k = 2. A separately released instruction-tuned version demonstrated that MoE LLMs can be adapted for downstream tasks via instruction tuning, reaching competitive benchmark performance while using fewer active parameters per token than a comparably performing dense model.

Beyond language, MoE has been applied to Vision Transformers (Vision MoE, 15B parameters) and diffusion models.

Design considerations

The OLMoE report (Muennighoff et al., 2024) documents that numerous design choices affect training stability and final performance: expert granularity (many small experts vs. few large ones), routing algorithm, capacity factor, auxiliary loss weighting, and the decision to use shared vs. purely routed experts. Static routing — where the assignment is computed by a fixed hash function or random seed rather than a learned gate — has also been studied as a way to remove the routing from the optimization loop entirely. There is no consensus best configuration; the space remains actively explored.