#llama-cpp
Wiki 5
- 80 tok/s + 128K Context on 12GB VRAM — Qwen3.6 + MTP janvitos's RTX 4070 Super config — Qwen3.6-35B-A3B MTP UD-Q4_K_XL on 12GB via `-fitt 1536`; thread also documents turboquant+MTP on a GTX 1070
- Coding-Agent Tests of Local LLMs on 16 GB VRAM Vyacheslav's RTX 5070 Ti tests of Gemma 4 / Qwen 3.6 / Qwen Coder via OpenCode + llama.cpp — Gemma wins 12/12, Thinking mode hurts rule-following
- Local LLMs Deep Dive — Quants, MoE Offload, REAP, ik_llama, Speculative Decoding Vyacheslav's long-form Habr primer covering BF16, K/I-quants, dynamic quantization, `-fit`/`-cmoe`/`-ncmoe`, REAP expert pruning, ik_llama, MTP/EAGLE3, Linux vs Windows
- MoE CPU Offload (`--n-cpu-moe`) llama.cpp's MoE-aware offload — keep attention in VRAM, push inactive expert weights to system RAM via PCIe; +55-60% over naïve layer-based offload
- Speculative Decoding Predict multiple tokens per forward pass, verify in batch — four llama.cpp variants (draft model, MTP, EAGLE3, ngram-mod) trade off draft quality, memory, and applicability