Mixture of Experts (MoE)

Mixture-of-Experts is an LLM architecture where each layer contains many parallel subnetworks ("experts") and a small router selects which few to use for each token. Total parameter count and active parameter count are different numbers. A 35B-A3B model has 35 billion weights total but only ~3 billion participate in any single forward pass.

This split matters because it decouples model capacity from compute cost. The router and the active experts have to be in fast memory to keep generation responsive. The inactive experts can sit anywhere — system RAM, SSD, even network — as long as they reach the GPU before they're needed. Dense models can't do this; every weight participates in every token.

Why the headline number lies

Comparing MoE models by total parameters is misleading. Three models from Vyacheslav's tests:

Gemma 4 has the smallest total but the largest active count. In Vyacheslav's coding tests it also produced the best results, which suggests architecture and active-param budget matter more than total weights.

What MoE buys local inference

Two things, both critical for consumer hardware:

1. Lower compute per token — only the active experts run, so a 30B model generates at speeds closer to a 3B dense model.
2. Selective offload — inactive experts can live in slow memory without blocking generation. This is what makes llama.cpp --n-cpu-moe work; see moe-cpu-offload.

The cost is that all expert weights still have to be loaded somewhere. A 30B MoE doesn't fit in 16 GB VRAM at Q4 any more easily than a 30B dense model would — it just gives you a working way to spill the cold parts.

What MoE doesn't buy

• The router is not "smart" in a human sense. It learns which experts to activate during training; you don't get to pick or interpret them.
• Some MoE architectures interact badly with quantization formats. Unsloth's Dynamic 2.0 quants degrade more on Qwen MoE than on dense models because aggressive 4-bit compression collides with the router's sensitivity to small weight perturbations. See llm-quantization.
• Long sequences and large file edits stress MoE more than dense models because expert routing can shift unpredictably as context grows — one observed cause of Qwen's poor long-session behavior in local-llm-16gb-vram-tests.

Total params predict knowledge, not active params

Bojie Li's IKP paper (April 2026) regresses factual capacity against MoE parameter counts across 37 MoE models. Total parameters predict knowledge capacity (R² = 0.79); active parameters do not (R² = 0.51). Factual knowledge is stored across all expert weights, not only the ones routed to per token. This validates a piece of folk knowledge that hadn't had a clean empirical anchor: when you see "DeepSeek-V3 is a 671B model, 37B active," the 671B is the right number to compare against a 671B dense model for knowledge, even though the 37B is the right comparison for inference compute.

The corollary: REAP-pruning experts (reap-expert-pruning) trades knowledge capacity for size at a measurable rate. Cutting 20% of experts reduces both compute and the model's effective stored facts.

Notable MoE models in this wiki

• Gemma 4 26B-A4B — Google's MoE; the surprise winner of local-llm-16gb-vram-tests.
• Qwen 3.6 35B-A3B — Alibaba's reasoning generation.
• Qwen3-Coder 30B-A3B — code-specialized sibling.
• DeepSeek V3 / R1 / V4 — the line that proved MoE could be cost-effective at scale; see deepseek.