llama.cpp

Georgi Gerganov's C/C++ LLM inference engine. The reference implementation for running open-weights LLMs in the GGUF format on CPU and GPU. Most other local-LLM tools (Ollama, LM Studio, koboldcpp, llamafile) wrap it.

What it does

llama.cpp loads quantized model weights (GGUF format), runs forward passes, and serves an OpenAI-compatible HTTP API. The implementation is in plain C/C++ with backend drivers for CUDA, Metal, ROCm, Vulkan, SYCL, and CPU. Quantization formats — Q2, Q3, Q4, Q5, Q6, Q8 at various block layouts — are baked into the file format; you don't need a separate runtime to load Q4_K_M vs MXFP4.

The reason it's the reference implementation: Gerganov writes the architecture support for new models directly, so when a new model drops (Gemma 4, Qwen 3.6, DeepSeek V4), the first place it works is llama.cpp, and everything else inherits.

Why build from source

Pre-built wrappers (Ollama, LM Studio) ship generic binaries that target broad hardware compatibility. They miss architecture-specific optimizations. Vyacheslav reports up to 30% throughput loss running through LM Studio or Ollama compared to llama.cpp built locally against Blackwell tensor cores. On 16 GB VRAM where every token-per-second matters, that gap is the difference between usable and unusable.

Building from source also exposes flags that wrappers hide:

• --n-cpu-moe N — push expert blocks of N layers to system RAM; the key flag for fitting MoE models in low VRAM (see moe-cpu-offload)
• -fit / -fitt N — newer default that auto-detects MoE vs Dense and picks the offload mode for you, leaving N MB of VRAM free (see moe-cpu-offload)
• --cache-type-k, --cache-type-v — KV cache precision (q8_0, q4_0, f16); halving precision halves cost per token (see kv-cache-sizing); the llama-cpp-turboquant fork adds turbo4 / turbo3 for higher quality at the same byte budget (see random-rotation-quantization)
• --flash-attn 1 — Flash Attention; meaningful speedup on supported hardware
• --jinja — Jinja chat-template rendering for proper tool-call formats; required for agentic harnesses to see the tool list
• --reasoning off, --reasoning-budget 0 — toggle reasoning-mode generation (insufficient alone for Qwen — see thinking-mode-rule-erosion)
• --spec-type mtp|eagle3|ngram-mod|draft — speculative decoding (PR #22673 for MTP, #18039 for EAGLE3, #19164 for ngram-mod); see speculative-decoding

Architecture-aware MoE offload

The headline operational feature for low-VRAM users is --n-cpu-moe. It splits a Mixture-of-Experts model on the architecture-aware axis: attention layers stay in VRAM, inactive expert blocks spill to RAM, only fetched over PCIe when the router selects them. This gives 50-60% throughput gains over the naïve --n-gpu-layers offload on Qwen 3.6 in local-llm-16gb-vram-tests.

Basic server usage

llama-server \
  --model "gemma-4-26B-A4B-it-MXFP4_MOE.gguf" \
  --ctx-size 32768 \
  --n-gpu-layers 999 \
  --n-cpu-moe 8 \
  --flash-attn 1 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --batch-size 512 \
  --ubatch-size 512 \
  --reasoning off \
  --jinja \
  --temp 0.7 \
  --top-p 0.8 \
  --top-k 20 \
  --no-mmap

Serves an OpenAI-compatible API on localhost:8080. Agentic harnesses (OpenCode, Cline, Roo Code, etc.) point at it as a custom provider.

Limitations

• No tool-parameter streaming in the OpenAI-style endpoint by default — tool calls buffer and emit atomically. This is one of the local-LLM UX gaps Ronacher complains about and the reason coding agents on llama.cpp feel jerkier than on hosted providers.
• Tuning is empirical. There is no "auto" configuration that picks --n-cpu-moe, batch sizes, or quant choice for you. The community-maintained MoE models all want slightly different settings.
• No first-class support for non-text modalities — image and audio LLMs need llava-cpp, whisper.cpp, or model-specific forks.

Related

• mixture-of-experts — architecture llama.cpp's MoE features target
• llm-quantization — Q4_K_M, MXFP4, UD-Q4_K_XL formats llama.cpp loads
• OpenCode — agent harness that pairs cleanly with llama.cpp
• lucumr-local-models — Ronacher's case for ds4.c-style narrow alternatives over generic GGUF runners

Repo: https://github.com/ggml-org/llama.cpp · ~96k stars · MIT