MoE CPU Offload (`--n-cpu-moe`)

--n-cpu-moe is a llama.cpp flag that splits a Mixture-of-Experts model along the architecture-aware axis: attention layers stay on the GPU for every token, but expert blocks can be assigned to system RAM. The router fires only a few experts per token, so only those experts have to travel over PCIe — the bus traffic is small enough that the GPU doesn't stall waiting for weights.

This works much better than the classic --n-gpu-layers offload, which has no notion of MoE structure and treats every layer as monolithic. Layer-based offload puts whole layers (attention + all experts in that block) on either GPU or CPU, so an expert you actually need for the next token might be in RAM, and a router decision can stall on a full PCIe round-trip.

Measured impact

From Vyacheslav's tests on RTX 5070 Ti 16 GB (local-llm-16gb-vram-tests):

The size of the gain tracks how much of the model has to spill. Qwen 3.6 (35B total) spills more than Qwen Coder (30B total), so it sees a bigger win when the spill is structured intelligently. Gemma 4 fits fully in VRAM and doesn't need offload.

How it works in practice

Two flags interact:

• --n-gpu-layers 99 (or 999) — load every layer onto the GPU, conceptually. With dense models this would just OOM; with MoE the next flag rescues it.
• --n-cpu-moe N — push the expert blocks of N layers off the GPU into RAM.

N is tuned empirically: start somewhere, watch VRAM with nvtop or similar, raise N if you OOM under load, lower it if you have headroom. Vyacheslav's calibration loop: 14.0-15.3 GB VRAM at peak is the target zone, leaving 1-1.5 GB free for KV cache spikes.

Typical settings from the article:

The Qwen models need more spill because they have more total weight. Going to 256K context costs another 6 units on Qwen, 4 units on Gemma (Gemma's KV cache is ~4× larger per token, so more of its budget goes to KV; see kv-cache-sizing).

What this enables

This is what makes the Habr article's coding tests possible at all. Without --n-cpu-moe, the 30B and 35B models would either not fit or generate too slowly to be usable. With it, they generate at 60+ tokens/sec on a $1000-ish consumer card.

The broader point — and the one Armin Ronacher is complaining about — is that this knob exists, matters enormously, and is invisible to anyone running through Ollama or LM Studio. Those convenience wrappers don't expose it. Building llama.cpp from source and learning the flag is roughly a +50% performance upgrade on the right model class. That's the "polish gap" Ronacher writes about.

The full mode family

llama.cpp now ships four related flags. From Vyacheslav's primer and the Reddit thread:

-fit is the new default. It detects MoE architecture and uses -ncmoe automatically; for Dense it uses -ngl. You can override with -cmoe to force full CPU offload of experts. --fit-target 1024 reserves a specific amount of free VRAM (-fitt 1536 is the explicit form of -fit with target 1536 MB).

Practical implication: the earlier tests article specified --n-cpu-moe 17 manually. With -fit, the manual tuning is unnecessary on current llama.cpp — pick --fit-target and let it choose. The Reddit setup uses -fitt 1536 to leave room for MTP draft heads plus 128K KV cache.

Why Ollama doesn't have this

Ollama uses llama.cpp's GGML engine under the hood, so it could enable -ncmoe for MoE models. It doesn't. It uses -ngl for everything, including MoE. The result on RTX 4060 16GB with Qwen3.6-35B-A3B Q4_K_M (Vyacheslav's measurement):

The 3× difference is entirely the mode choice, not the engine. When Ollama enables -ncmoe, the gap closes.

Cross-references

• mixture-of-experts — why this is architecturally possible
• kv-cache-sizing — the other half of the VRAM budget
• speculative-decoding — combines with this; -fitt budget has to cover draft heads too
• llama.cpp — the inference engine that implements this