What is mixture of experts (MoE) in LLMs?

Also called MoE, sparse mixture of experts, sparse MoE

Definition

Mixture of experts (MoE) is a neural network architecture in which a learned router sends each token to a few specialized subnetworks, called experts, so only part of the model runs per token.

A mixture of experts model replaces some dense layers with a set of parallel expert networks plus a router, also called a gating network, that selects which experts process each input. The idea dates to work by Jacobs, Jordan, Nowlan, and Hinton in 1991. Noam Shazeer and colleagues at Google brought it to modern deep learning with the sparsely-gated mixture-of-experts layer in 2017, reporting capacity gains of more than 1,000 times with only minor losses in computational efficiency. Open-weight large language models including Mixtral 8x7B and DeepSeek-V3 use the architecture.

How mixture of experts works#

In an MoE transformer, the feed-forward block in each layer is replaced by several expert feed-forward networks. For every token, the router scores the experts, sends the token to the top-scoring few, and combines their outputs using the router's weights.

Mixtral 8x7B, released by Mistral AI in December 2023, is a well-documented example: 8 experts per layer, 2 selected per token, 46.7B total parameters, and 12.9B parameters used per token. DeepSeek-V3 has 671B total parameters with 37B activated per token. MoE training also needs a load-balancing mechanism, often an auxiliary loss, so the router does not overload a few experts while others sit idle.

Mixture of experts vs dense models#

A dense model runs every parameter for every token. An MoE model's compute per token scales with its active parameters, while its capacity scales with its total parameters.

Dense model MoE model
Parameters used per token All Active subset (Mixtral: 12.9B of 46.7B)
Compute per token Scales with total size Scales with active size
Memory to serve All weights All weights, since every expert must be loaded
Serving complexity Lower Higher: expert parallelism, uneven routing

The memory row is the catch. An MoE model needs enough GPU memory for every expert even though each token uses only a few, which is why quantization and multi-GPU serving matter for MoE deployment (see the LLM inference optimization guide). When comparing models on LLM benchmarks, check both total and active parameter counts.

Cite this definition

The AI Engineering Editors. "What is mixture of experts (MoE) in LLMs?" The AI Engineering, updated Nov 06, 2026. https://theaiengineering.com/glossary/mixture-of-experts

Title
What is mixture of experts (MoE) in LLMs?
Author
The AI Engineering Editors
Publisher
The AI Engineering
Updated
View as Markdown

Part of the LLM models and benchmarks hub · Updated