# MoE CPU Offload (`--n-cpu-moe`)

`--n-cpu-moe` is a llama.cpp flag that splits a Mixture-of-Experts model along the architecture-aware axis: attention layers stay on the GPU for every token, but expert blocks can be assigned to system RAM. The router fires only a few experts per token, so only those experts have to travel over PCIe — the bus traffic is small enough that the GPU doesn't stall waiting for weights.

This works much better than the classic `--n-gpu-layers` offload, which has no notion of MoE structure and treats every layer as monolithic. Layer-based offload puts whole layers (attention + all experts in that block) on either GPU or CPU, so an expert you actually need for the next token might be in RAM, and a router decision can stall on a full PCIe round-trip.

## Measured impact

From Vyacheslav's tests on RTX 5070 Ti 16 GB ([[local-llm-16gb-vram-tests]]):

| Model + mode | `--n-gpu-layers` | `--n-cpu-moe` | Gain |
| --- | --- | --- | --- |
| Qwen 3.6 Fast | 41.3 t/s | **64.0** | +55% |
| Qwen 3.6 Thinking | 41.7 t/s | **66.7** | +60% |
| Qwen Coder Fast | 51.3 t/s | **57.6** | +12% |
| Qwen Coder Thinking | 53.0 t/s | **60.2** | +14% |

The size of the gain tracks how much of the model has to spill. Qwen 3.6 (35B total) spills more than Qwen Coder (30B total), so it sees a bigger win when the spill is structured intelligently. Gemma 4 fits fully in VRAM and doesn't need offload.

## How it works in practice

Two flags interact:

- `--n-gpu-layers 99` (or 999) — load every layer onto the GPU, conceptually. With dense models this would just OOM; with MoE the next flag rescues it.
- `--n-cpu-moe N` — push the expert blocks of N layers off the GPU into RAM.

`N` is tuned empirically: start somewhere, watch VRAM with nvtop or similar, raise `N` if you OOM under load, lower it if you have headroom. Vyacheslav's calibration loop: 14.0-15.3 GB VRAM at peak is the target zone, leaving 1-1.5 GB free for KV cache spikes.

Typical settings from the article:

| Model | `--n-cpu-moe` (32K ctx) | `--n-cpu-moe` (256K ctx, KV q4_0) |
| --- | --- | --- |
| Gemma 4 26B | 8 | 12 |
| Qwen 3.6 35B | 17 | 23 |
| Qwen Coder 30B | 17 | — |

The Qwen models need more spill because they have more total weight. Going to 256K context costs another 6 units on Qwen, 4 units on Gemma (Gemma's KV cache is ~4× larger per token, so more of its budget goes to KV; see [[kv-cache-sizing]]).

## What this enables

This is what makes the [[local-llm-16gb-vram-tests|Habr article's coding tests]] possible at all. Without `--n-cpu-moe`, the 30B and 35B models would either not fit or generate too slowly to be usable. With it, they generate at 60+ tokens/sec on a $1000-ish consumer card.

The broader point — and the one [[lucumr-local-models|Armin Ronacher]] is complaining about — is that this knob exists, matters enormously, and is invisible to anyone running through Ollama or LM Studio. Those convenience wrappers don't expose it. Building llama.cpp from source and learning the flag is roughly a +50% performance upgrade on the right model class. That's the "polish gap" Ronacher writes about.

## The full mode family

llama.cpp now ships four related flags. From [[habr-local-llm-quantization-deep-dive|Vyacheslav's primer]] and the [[reddit-qwen3-6-mtp-12gb|Reddit thread]]:

| Flag | What it does | Use case |
| --- | --- | --- |
| `-ngl N` | Load N layers on GPU, rest on CPU | Dense models |
| `-ncmoe N` | Put expert blocks of N MoE layers on CPU | MoE, partial spill |
| `-cmoe` | All MoE expert weights on CPU | MoE, minimum VRAM |
| `-fit` (default on) | Auto-detect MoE vs Dense, pick mode, leave ~1 GB free | Most users |
| `-fitt N` | Like `-fit` but leave N MB free | When you need explicit headroom for KV cache or draft model |

`-fit` is the new default. It detects MoE architecture and uses `-ncmoe` automatically; for Dense it uses `-ngl`. You can override with `-cmoe` to force full CPU offload of experts. `--fit-target 1024` reserves a specific amount of free VRAM (`-fitt 1536` is the explicit form of `-fit` with target 1536 MB).

Practical implication: the [[local-llm-16gb-vram-tests|earlier tests article]] specified `--n-cpu-moe 17` manually. With `-fit`, the manual tuning is unnecessary on current llama.cpp — pick `--fit-target` and let it choose. The Reddit setup uses `-fitt 1536` to leave room for MTP draft heads plus 128K KV cache.

## Why Ollama doesn't have this

Ollama uses llama.cpp's GGML engine under the hood, so it *could* enable `-ncmoe` for MoE models. It doesn't. It uses `-ngl` for everything, including MoE. The result on RTX 4060 16GB with Qwen3.6-35B-A3B Q4_K_M ([[habr-local-llm-quantization-deep-dive|Vyacheslav's measurement]]):

| Engine | Mode | Speed at 4K context | Speed at 32K context |
| --- | --- | --- | --- |
| Ollama | `-ngl` | 16 t/s | 11 t/s |
| llama.cpp | `-fit` (auto `-ncmoe`) | **45 t/s** | **36 t/s** |

The 3× difference is entirely the mode choice, not the engine. When Ollama enables `-ncmoe`, the gap closes.

## Cross-references

- [[mixture-of-experts]] — why this is architecturally possible
- [[kv-cache-sizing]] — the other half of the VRAM budget
- [[speculative-decoding]] — combines with this; `-fitt` budget has to cover draft heads too
- [[toolbox/llama-cpp|llama.cpp]] — the inference engine that implements this