# llama.cpp

Georgi Gerganov's C/C++ LLM inference engine. The reference implementation for running open-weights LLMs in the GGUF format on CPU and GPU. Most other local-LLM tools (Ollama, LM Studio, koboldcpp, llamafile) wrap it.

## What it does

llama.cpp loads quantized model weights (GGUF format), runs forward passes, and serves an OpenAI-compatible HTTP API. The implementation is in plain C/C++ with backend drivers for CUDA, Metal, ROCm, Vulkan, SYCL, and CPU. Quantization formats — Q2, Q3, Q4, Q5, Q6, Q8 at various block layouts — are baked into the file format; you don't need a separate runtime to load Q4_K_M vs MXFP4.

The reason it's the reference implementation: Gerganov writes the architecture support for new models directly, so when a new model drops (Gemma 4, Qwen 3.6, DeepSeek V4), the first place it works is llama.cpp, and everything else inherits.

## Why build from source

Pre-built wrappers (Ollama, LM Studio) ship generic binaries that target broad hardware compatibility. They miss architecture-specific optimizations. Vyacheslav reports up to **30% throughput loss** running through LM Studio or Ollama compared to llama.cpp built locally against Blackwell tensor cores. On 16 GB VRAM where every token-per-second matters, that gap is the difference between usable and unusable.

Building from source also exposes flags that wrappers hide:

- `--n-cpu-moe N` — push expert blocks of N layers to system RAM; the key flag for fitting MoE models in low VRAM (see [[moe-cpu-offload]])
- `-fit` / `-fitt N` — newer default that auto-detects MoE vs Dense and picks the offload mode for you, leaving N MB of VRAM free (see [[moe-cpu-offload]])
- `--cache-type-k`, `--cache-type-v` — KV cache precision (q8_0, q4_0, f16); halving precision halves cost per token (see [[kv-cache-sizing]]); the `llama-cpp-turboquant` fork adds `turbo4` / `turbo3` for higher quality at the same byte budget (see [[random-rotation-quantization]])
- `--flash-attn 1` — Flash Attention; meaningful speedup on supported hardware
- `--jinja` — Jinja chat-template rendering for proper tool-call formats; *required* for agentic harnesses to see the tool list
- `--reasoning off`, `--reasoning-budget 0` — toggle reasoning-mode generation (insufficient alone for Qwen — see [[thinking-mode-rule-erosion]])
- `--spec-type mtp|eagle3|ngram-mod|draft` — speculative decoding (PR #22673 for MTP, #18039 for EAGLE3, #19164 for ngram-mod); see [[speculative-decoding]]

## Architecture-aware MoE offload

The headline operational feature for low-VRAM users is `--n-cpu-moe`. It splits a Mixture-of-Experts model on the architecture-aware axis: attention layers stay in VRAM, inactive expert blocks spill to RAM, only fetched over PCIe when the router selects them. This gives 50-60% throughput gains over the naïve `--n-gpu-layers` offload on Qwen 3.6 in [[local-llm-16gb-vram-tests]].

## Basic server usage

```bash
llama-server \
  --model "gemma-4-26B-A4B-it-MXFP4_MOE.gguf" \
  --ctx-size 32768 \
  --n-gpu-layers 999 \
  --n-cpu-moe 8 \
  --flash-attn 1 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --batch-size 512 \
  --ubatch-size 512 \
  --reasoning off \
  --jinja \
  --temp 0.7 \
  --top-p 0.8 \
  --top-k 20 \
  --no-mmap
```

Serves an OpenAI-compatible API on `localhost:8080`. Agentic harnesses (OpenCode, Cline, Roo Code, etc.) point at it as a custom provider.

## Limitations

- No tool-parameter streaming in the OpenAI-style endpoint by default — tool calls buffer and emit atomically. This is one of the local-LLM UX gaps [[lucumr-local-models|Ronacher complains about]] and the reason coding agents on llama.cpp feel jerkier than on hosted providers.
- Tuning is empirical. There is no "auto" configuration that picks `--n-cpu-moe`, batch sizes, or quant choice for you. The community-maintained [[mixture-of-experts|MoE]] models all want slightly different settings.
- No first-class support for non-text modalities — image and audio LLMs need llava-cpp, whisper.cpp, or model-specific forks.

## Related

- [[mixture-of-experts]] — architecture llama.cpp's MoE features target
- [[llm-quantization]] — Q4_K_M, MXFP4, UD-Q4_K_XL formats llama.cpp loads
- [[toolbox/opencode|OpenCode]] — agent harness that pairs cleanly with llama.cpp
- [[lucumr-local-models]] — Ronacher's case for `ds4.c`-style narrow alternatives over generic GGUF runners

Repo: https://github.com/ggml-org/llama.cpp · ~96k stars · MIT