REAP — Router-weighted Expert Activation Pruning

Router-weighted Expert Activation Pruning is a structural compression technique for Mixture-of-Experts models. Instead of quantizing the weights of all experts harder (the quantization route), REAP removes whole experts — the ones the router rarely activates for the workloads in a calibration dataset. The remaining experts cover whatever specializations the calibration data exercised; experts trained for unused specializations are gone.

This works because MoE experts have non-uniform usage. Some experts specialize narrowly — e.g., one might handle leap-year tables, another handles minor European languages. For a given downstream workload (e.g., programming), most narrow experts are idle. They occupy disk, RAM, and bandwidth without doing work. REAP measures activation patterns on the calibration data, ranks experts by router weight, and cuts the bottom N%.

What the numbers look like

A REAP-20 cut on Qwen 3.6 (35B-A3B, 256 experts, 8 active) becomes 28B-A3B (about 20% smaller, same active-parameter count). REAP-50 and REAP-75 variants exist for tighter budgets. The repo for these is huggingface.co/models?search=reap — most published REAP versions are calibrated on programming-focused datasets.

Vyacheslav's worked example (habr-local-llm-quantization-deep-dive): Qwen3.6-28B-REAP20-A3B Q4_K_M successfully extends the Minecraft clone he previously built with the full Qwen3.6 — added caves, diamond placement, a generated house. The agentic loop survives. The Russian-language quality degrades noticeably (the calibration was English-coding-focused, so Russian capabilities fade first — REAP "forgets who Chapaev is"). For English programming tasks the cut is nearly invisible.

The same technique applied to large MoE models is more interesting because the absolute size savings matter: GLM-5.1 744B-A40B → 444B-A14B saves ~40% of disk and active-compute. At the trillion-parameter scale, REAP becomes a path to "fit a frontier-class MoE into your RAM" without changing quantization.

Caveats

• The calibration data is the model's new behavioral envelope. Cut 20% of experts to specialize for coding, and the model is genuinely worse at non-coding tasks. There's no free 20%.
• Dynamic quantization is rarely available for REAP versions yet. The Qwen3.6-28B-REAP20-A3B published as of mid-2026 ships only as Q4_K_M; UD-Q4_K_XL versions exist for the unpruned model but not the REAP variant. So combining REAP with the best quant recipe is still future work.
• The cut targets the router output, which means rare-but-essential experts can be eliminated. The standard imatrix-style argument — that activation frequency tracks importance — assumes the calibration dataset covers the target distribution. If your real workload draws on an expert the calibration dataset didn't exercise, you lose that capability silently.
• REAP composes with quantization but the order matters. The published versions are quantized after pruning. Quantizing first and then pruning is a different operation that nobody has published results on.

How it compares to alternatives

REAP is the MoE-specific structural pruning analogue. It is also a relatively new technique — Vyacheslav notes that as of mid-2026 it's "not very widespread" and the published variants are mostly Qwen and GLM.

Cross-references

• mixture-of-experts — the architecture REAP targets
• llm-quantization — the alternative compression route, often combined with REAP
• habr-local-llm-quantization-deep-dive — Vyacheslav's worked example with the Minecraft clone
• moe-cpu-offload — the operational complement: REAP shrinks total weight; cpu-moe spills the rest