#moe
Wiki 6
- 80 tok/s + 128K Context on 12GB VRAM — Qwen3.6 + MTP janvitos's RTX 4070 Super config — Qwen3.6-35B-A3B MTP UD-Q4_K_XL on 12GB via `-fitt 1536`; thread also documents turboquant+MTP on a GTX 1070
- Coding-Agent Tests of Local LLMs on 16 GB VRAM Vyacheslav's RTX 5070 Ti tests of Gemma 4 / Qwen 3.6 / Qwen Coder via OpenCode + llama.cpp — Gemma wins 12/12, Thinking mode hurts rule-following
- Kimi K3 Raschka on Moonshot's 2.8T open-weight K3 — LatentMoE, KDA, attention residuals, NoPE everywhere
- Local LLMs Deep Dive — Quants, MoE Offload, REAP, ik_llama, Speculative Decoding Vyacheslav's long-form Habr primer covering BF16, K/I-quants, dynamic quantization, `-fit`/`-cmoe`/`-ncmoe`, REAP expert pruning, ik_llama, MTP/EAGLE3, Linux vs Windows
- MoE CPU Offload (`--n-cpu-moe`) llama.cpp's MoE-aware offload — keep attention in VRAM, push inactive expert weights to system RAM via PCIe; +55-60% over naïve layer-based offload
- REAP — Router-weighted Expert Activation Pruning Cut the least-active experts from an MoE model based on a target-domain calibration dataset; 20% size reduction with most agentic-coding capability retained
Toolbox 1
- deltafin Runs Kimi K3's 2.8T weights on one Apple Silicon Mac, streaming MXFP4 experts over HTTP to disk