Mixture-of-Experts (MoE)

Definition

Mixture-of-Experts (MoE) is the architecture behind every model card that reads like "671B total parameters, 37B active." Instead of one feed-forward network per transformer layer, an MoE layer holds hundreds of parallel copies of it — the experts — plus a small router that picks a handful of them for each token. Compute is billed against the experts that fire. Memory is billed against all of them.

That split is the whole idea and the whole trap. DeepSeek-V3's paper states plainly that it is a model "with 671B total parameters with 37B activated for each token." Only 5.5% of the model does arithmetic for any given token, so it costs about as much to run as a 37-billion-parameter dense model — roughly 18 times less arithmetic than a dense model its own size. But every one of those 671 billion parameters has to be sitting in fast memory when the router asks for it, so it costs about as much to host as a dense 671B model. A team that reads "37B active", buys one 80 GB GPU and expects the weights to fit has under-provisioned by a factor of eighteen.

The modern form of the idea arrived in 2017, under an unusually honest title: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. It stacked "up to thousands of feed-forward sub-networks" into a language model of 137 billion parameters, at a time when a large model had under one billion, and reported "greater than 1000x improvements in model capacity with only minor losses in computational efficiency." Every MoE shipped since is a variation on that same trade: buy parameters, which are cheap in memory, instead of FLOPs, which are expensive in time.

How It Works

Where the parameters actually are

A transformer layer is attention followed by a feed-forward network. Attention is a small share of the weights; the feed-forward block is the bulk. MoE replicates only the feed-forward block, and leaves attention shared.

DeepSeek-V3 publishes its configuration, so the arithmetic can be done from the actual numbers: 61 layers, of which the first 3 are left dense; hidden size 7,168; expert intermediate size 2,048; 256 routed experts plus 1 always-on shared expert per MoE layer; 8 routed experts selected per token.

One expert is three matrices of 7,168 x 2,048 — the gate, up and down projections — which is 44.0 million parameters. Two hundred and fifty-seven of them per layer is 11.3 billion, and across the 58 MoE layers that is 656.5 billion parameters, or 98% of the model. So "671B total" is very nearly a statement about how many copies of one small feed-forward block the checkpoint contains.

Now run the same sum over what actually fires: 8 routed experts plus the 1 shared expert is 9 of 257, which is 396 million parameters per layer and 23.0 billion across the 58 MoE layers. The remaining ~14 billion of the advertised 37B active is attention, embeddings and the three dense layers, all of which run for every token regardless of what the router decides. The model card reconciles exactly, and it reconciles into two very different quantities.

The router, and why k is small

Routing is done by a single matrix per MoE layer, of size hidden x experts — 7,168 x 256, about 1.8 million parameters, or 106 million across the whole model. That is 0.016% of the checkpoint deciding what the other 98% does. It scores every expert against the token's hidden state, keeps the top k, normalises those k scores into weights, and sums the selected experts' outputs weighted accordingly. The other experts are not computed and discarded; they are never computed.

Cost is roughly linear in k, so k is kept small. Mixtral 8x7B, the model that made open-weight MoE ordinary, used 8 experts and picked 2: "Mixtral has 46.7B total parameters but only uses 12.9B parameters per token." Google's Switch Transformer went further and picked one — "we instead use a simplified strategy where we route to only a single expert" — reporting "up to 7x increases in pre-training speed" and scaling to a 1.6-trillion-parameter model across 2,048 experts. The prior worry had been that k=1 leaves the gating network without a useful gradient, since there is nothing to weigh the chosen expert against. Switch showed it trains anyway.

The later move was not to raise k as a fraction but to raise it as a count, by making each expert smaller. Databricks put the argument in one sentence: "DBRX has 16 experts and chooses 4, while Mixtral and Grok-1 have 8 experts and choose 2. This provides 65x more possible combinations of experts and we found that this improves model quality." The 65 is not a slogan — it is 1,820 divided by 28. Push the same logic to DeepSeek-V3's 8-of-256 and the number of distinct expert combinations is about 4.1 x 10^14, for the same active-parameter budget. This is what "fine-grained MoE" means: more, smaller specialists, and far more ways to combine them.

The shared expert that sits alongside them is the other half of that design. Some knowledge every token needs — basic syntax, common word senses — and without a shared expert each of the 256 routed experts has to learn its own redundant copy. Making one expert always-on lets the routed ones spend their capacity on what actually distinguishes them.

Load balancing, and what happens without it

Routing is a positive feedback loop. An expert that wins slightly more tokens early receives more gradient, improves faster, and therefore wins more tokens. Left alone, training collapses onto a small subset of experts while the rest drift near their initialisation. The result is the worst of both worlds: the compute savings of a sparse model with the memory bill of a large dense one, most of it paid for parameters that never contribute anything. This is expert collapse, and it is the failure mode every MoE training recipe is built around.

The standard fix is an auxiliary loss added to the training objective. Switch Transformer's version is loss = α · N · Σ fᵢ · Pᵢ, where fᵢ is the fraction of tokens routed to expert i and Pᵢ is the router's mean probability for it, with the coefficient α set to 10⁻² — "sufficiently large to ensure load balancing while small enough to not overwhelm the primary cross-entropy objective." The product is minimised when both distributions are uniform, at which point the term contributes nothing beyond α itself.

The catch is that this is a second objective competing with the one you care about: the model is being pushed to spread tokens evenly even when the linguistically correct routing is uneven. DeepSeek-V3 attacks that directly, saying it "pioneer[s] an auxiliary-loss-free strategy for load balancing, which minimizes the performance degradation that arises from encouraging load balance." Instead of a gradient term, a per-expert bias is nudged up or down between steps according to observed load, changing which experts win the top-k without ever entering the loss.

Token-choice, expert-choice, and dropped tokens

Everything above is token-choice routing: each token picks its k experts. Because experts are given fixed-size buffers on each device, a popular expert can overflow, and the tokens that do not fit are dropped — they skip the expert layer entirely and pass through the residual connection unchanged. The buffer size relative to the average load is the capacity factor, and it trades wasted memory against dropped tokens.

Expert-choice routing inverts the assignment: rather than tokens selecting experts, it has "experts selecting the top-k tokens." Each expert takes a fixed number of tokens, so load is balanced by construction with no auxiliary loss and no dropping, while a token may end up processed by many experts or none. The paper reports that this "improves training convergence time by more than 2x" against Switch-style top-1 and GShard-style top-2 gating. The cost is that an expert must see a whole batch before choosing, which is natural in training and awkward in autoregressive decoding.

One more distinction is worth naming only to dismiss it. A dense MoE runs every expert and combines all their outputs; it costs the full parameter count in FLOPs and buys nothing but an ensembling effect. Every MoE in a shipping language model is sparse, and when practitioners say "MoE" they mean sparse MoE.

The arithmetic that decides your bill

A forward pass costs roughly 2 FLOPs per active parameter per token — one multiply and one add. For DeepSeek-V3 that is 2 x 37 billion = 74 GFLOPs per token, against 2 x 671 billion = 1,342 GFLOPs per token for a dense model of the same size. An 18x saving, exactly the ratio of total to active.

Memory follows the other number. At bfloat16, two bytes per parameter, 671 billion parameters is 1,342 GB of weights before a single byte of KV cache or activation. On 80 GB H100s that is 17 GPUs of pure weights; a standard eight-GPU node holds 640 GB, so two entire nodes are not enough. Halving the precision to FP8 via quantization brings it to 671 GB — still more than one node. A dense 37B model, with identical FLOPs per token, is 74 GB and fits on a single H100.

The two figures are the same size in different units because bfloat16 uses 2 bytes per parameter and a forward pass uses 2 FLOPs per parameter. That is the sentence worth keeping:

Active parameters set your latency and your compute bill. Total parameters set your memory bill, and therefore how many GPUs you must rent before you serve a single token.

Real-World Applications

MoE is now the default architecture for large open-weight models, and the published configurations show the direction of travel clearly.

Read down the last column and the trend is unmistakable. Mixtral fired 25% of its experts per token; DBRX 25%; DeepSeek-V3 3.1%; Kimi K3, which Moonshot describes as "effectively activating 16 out of 896 experts", about 1.8%. The industry did not decide that sparser is universally better — it discovered that at a fixed active-parameter budget, more and smaller experts are worth more than fewer and larger ones.

This changes real procurement decisions. Tencent's Hy3 is "a 295B-parameter Mixture-of-Experts (MoE) model" with "21B active parameters" and "192 experts, top-8 activated". At bfloat16 its weights are 590 GB, which fits inside a single 640 GB eight-GPU node with roughly 50 GB left over for KV cache and activations. DeepSeek-V3's 1,342 GB does not, and needs three such nodes. Both models advertise roughly 20-40 billion active parameters and comparable per-token latency; the thing that separates them for anyone self-hosting is the total, not the active count. Inference optimization work on an MoE almost always starts there.

The same asymmetry explains the API pricing that puts very large MoE models below much smaller dense ones. A provider serving thousands of concurrent requests amortises the fixed memory cost across all of them, and then charges per token — a quantity governed by the active parameters. MiniMax's model card describes M3 as having "~428B parameters and ~23B activated parameters"; its published config.json shows 128 routed experts with 4 picked per token plus a shared expert. The nameplate number is the one a self-hoster pays for and an API user never sees.

Key Concepts

Sparsity ratio. Active divided by total: 5.5% for DeepSeek-V3, 5.4% for MiniMax-M3, 7.1% for Tencent Hy3, and 27.6% for the much older Mixtral. It is the single number that tells you how far apart your compute bill and your memory bill will be.

Granularity. How finely the same active budget is chopped up. Running one expert of size 4S and running four of size S cost the same per token, but the second arrangement gives the router a choice. DBRX's 4-of-16 has 1,820 possible selections; a 1-of-4 layer with the same active budget has four.

Expert parallelism and the all-to-all. Because the experts do not fit on one GPU, they are spread across devices, and every MoE layer therefore has to ship each token's hidden state to whichever GPUs hold its chosen experts and collect the results back. That is two all-to-all collectives per MoE layer — 116 of them per token step for a 58-layer model like DeepSeek-V3 — and on a poorly connected cluster the network, not the GPUs, sets your throughput.

Challenges

Provisioning against the wrong number. This is the expensive one, and it is entirely avoidable. Reading "37B active" and sizing the cluster like a 37B dense model understates the memory requirement by 18x — 74 GB versus 1,342 GB. Sparse activation reduces the arithmetic per token; it does nothing whatsoever to the number of bytes that must be resident, because the router may send the very next token anywhere.

Comparing against the wrong baseline. MoE is faster than a dense model of the same total size, by roughly the sparsity ratio, and this is where the marketing numbers come from. Against a dense model of the same active size it is not faster at all: the same 74 GB of weights are read per token, the same 74 GFLOPs are performed, and the MoE adds routing and cross-device communication on top. Single-stream decoding is bound by memory bandwidth rather than arithmetic, so a "we cut FLOPs 18x" claim does not translate into an 18x latency win unless the baseline was genuinely a dense model of the full size.

Batching fights sparsity. A dense model turns a batch of 64 tokens into one tall matrix multiply. An MoE has to split that batch 256 ways. With perfectly uniform routing, a batch of 64 tokens choosing 8 of 256 experts touches about 222 distinct experts — 87% of them — so you read 87% of the expert weights to serve 64 tokens, and each expert receives an average of 2.3 tokens. Every one of those 222 matrix multiplies is a skinny, low-utilisation GEMM. That is why measured MoE throughput lands well short of what the FLOP ratio promises, and why serving stacks invest so heavily in grouped GEMMs, expert-parallel scheduling and capacity limits. Note that this is the balanced case; imbalance makes it worse, not better.

Fine-tuning perturbs the routing. A dense model fine-tuned on a narrow corpus adjusts all of its weights a little. An MoE fine-tuned on the same corpus also changes which experts see which tokens, and a distribution that was balanced during pre-training can skew badly on a specialised dataset — concentrating updates in a few experts while others go untouched. Balance is a property of the data as much as of the architecture.

Calibration data is spread thin. Because memory is the binding constraint, MoE models benefit disproportionately from quantization — but a calibration set that gives a dense layer a million tokens gives each of 256 experts a few thousand. Quantization error ends up uneven across experts, and the rarely-routed ones are exactly the ones with the least data to calibrate against.

Future Trends

Finer granularity, still. The active fraction has fallen from Mixtral's 2 of 8 to DeepSeek-V3's 8 of 256 to Kimi K3's 16 of 896, with total parameters rising the whole way. Nothing about the trade has changed — the field is simply discovering how far it goes before routing quality, all-to-all traffic or kernel efficiency becomes the binding constraint.

Balancing without a competing loss. DeepSeek-V3's bias-based, auxiliary-loss-free scheme removes a gradient term that was always fighting the language-modelling objective. Expect variants of it to displace the classic α-weighted balancing loss in new recipes.

Hardware-aware expert design. Expert shapes are increasingly chosen for what the GPU actually executes efficiently rather than for architectural tidiness. NVIDIA states that the Super and Ultra tiers of Nemotron 3 "utilize Latent MoE, a novel hardware-aware expert design for improved accuracy" — routing in a compressed space so that more experts fit the same inference budget.

Offload as a first-class deployment mode. The property that makes MoE awkward to batch makes it uniquely suited to tiered memory: since only a small fraction of experts is needed per token, local runtimes keep hot experts in VRAM and page cold ones from system RAM or NVMe. This is what puts 400B-class models on workstations rather than clusters, and it is the one place where a high total-to-active ratio is an advantage rather than a bill.

A different answer to the scaling laws. Dense scaling requires paying for every new parameter on every token. MoE decouples the two, letting capacity grow with memory — which is comparatively cheap and easy to add — while per-token compute grows far more slowly. As long as memory scales more cheaply than arithmetic, the incentive to keep widening that gap remains.

Code Example

The total and active counts on a model card are not marketing figures; they fall straight out of the published architecture. This derives both from DeepSeek-V3's config.json:

# DeepSeek-V3, from the values published in its config.json:
#   61 layers, the first 3 dense; hidden_size 7168; moe_intermediate_size 2048;
#   256 routed experts + 1 shared per MoE layer; 8 routed experts per token.
H, I = 7168, 2048
moe_layers = 61 - 3

expert = 3 * H * I                      # gate, up and down projections
per_layer_all    = expert * (256 + 1)   # every routed expert, plus the shared one
per_layer_active = expert * (8 + 1)     # the router's top-8, plus the shared one

print(f"one expert           {expert / 1e6:8.1f} M params")
print(f"all experts          {per_layer_all * moe_layers / 1e9:8.1f} B params")
print(f"experts per token    {per_layer_active * moe_layers / 1e9:8.1f} B params")
print(f"all routers          {H * 256 * moe_layers / 1e6:8.1f} M params")
print(f"weights at bf16      {671e9 * 2 / 1e9:8.0f} GB")
print(f"FLOPs per token      {2 * 37e9 / 1e9:8.0f} GFLOP")
print(f"FLOPs if dense       {2 * 671e9 / 1e9:8.0f} GFLOP")

Output:

one expert               44.0 M params
all experts             656.5 B params
experts per token        23.0 B params
all routers             106.4 M params
weights at bf16          1342 GB
FLOPs per token            74 GFLOP
FLOPs if dense           1342 GFLOP

Both headline figures fall out of four integers from a config file. The routers that steer all of it total 106 million parameters — one part in six thousand of the model.