#local-models

21 pages search “local-models” ↳ all tags

Wiki 15

  • 80 tok/s + 128K Context on 12GB VRAM — Qwen3.6 + MTP janvitos's RTX 4070 Super config — Qwen3.6-35B-A3B MTP UD-Q4_K_XL on 12GB via `-fitt 1536`; thread also documents turboquant+MTP on a GTX 1070
  • Alex Ellis Founder of OpenFaaS/Actuated/Inlets; blogs on self-hosted infra and local AI economics at blog.alexellis.io
  • Coding-Agent Tests of Local LLMs on 16 GB VRAM Vyacheslav's RTX 5070 Ti tests of Gemma 4 / Qwen 3.6 / Qwen Coder via OpenCode + llama.cpp — Gemma wins 12/12, Thinking mode hurts rule-following
  • Evaluating Quantized Models for Deployment ByteShape on why perplexity, KLD, and BPW don't rank quantized models for deployment
  • I Want Local Models to Work Armin Ronacher on why local-LLM UX is worse than it should be — fragmentation, missing tool-streaming, no critical mass — and his pi-ds4 / ds4.c bet
  • KV Cache Sizing KV cache memory per token = 2 × KV-heads × head-dim × layers × bytes/element; halving via q4_0 lets you double context for free
  • little-coder — Scaffold-Model Fit on Aider Polyglot Itay Inbarr shows Qwen3.5-9B Q4_K_M goes from 19.11% (default Aider) to 45.56% (little-coder) on Aider Polyglot by redesigning the scaffold around small-model behavior
  • LLM Quantization Formats Q4_K_M vs UD-Q4_K_XL vs MXFP4 — three approaches to 4-bit weight compression, with the perplexity gap that newer ≠ better
  • Local AI is not Opus Alex Ellis on why "local Qwen is near-Opus" is wrong, but a tuned local model still earns its keep in a small software business
  • Local LLMs Deep Dive — Quants, MoE Offload, REAP, ik_llama, Speculative Decoding Vyacheslav's long-form Habr primer covering BF16, K/I-quants, dynamic quantization, `-fit`/`-cmoe`/`-ncmoe`, REAP expert pruning, ik_llama, MTP/EAGLE3, Linux vs Windows
  • Mixture of Experts (MoE) Architecture where a router activates a small subset of "expert" subnetworks per token — splits total params from compute params and changes the local-inference economics
  • MoE CPU Offload (`--n-cpu-moe`) llama.cpp's MoE-aware offload — keep attention in VRAM, push inactive expert weights to system RAM via PCIe; +55-60% over naïve layer-based offload
  • Neutrino-1 8B Fermion Research's Apache-2.0 8B: a 3.88 GB ternary-coded container for H100, MacBook, and CPU
  • Speculative Decoding Predict multiple tokens per forward pass, verify in batch — four llama.cpp variants (draft model, MTP, EAGLE3, ngram-mod) trade off draft quality, memory, and applicability
  • Unsloth Quantization team behind the UD-Q*-XL GGUF family and Unsloth Dynamic 2.0 scheme; most-cited dynamic-quant publisher for Qwen/Gemma/DeepSeek

Toolbox 6

  • deltafin Runs Kimi K3's 2.8T weights on one Apple Silicon Mac, streaming MXFP4 experts over HTTP to disk
  • HARTOS Python AI runtime pitched as an OS — local inference, p2p federation, a closed learning core
  • ik_llama.cpp ikawrakow's llama.cpp fork with IQK quants, Q6 KV cache and faster long-context prompt processing
  • llama.cpp C/C++ inference engine for GGUF open-weights models, the base most local-LLM tools wrap
  • local-llm (jamesob) Build guide and configs for a 4x RTX PRO 6000 local inference rig, including PCIe switch tuning
  • OpenCode Provider-agnostic terminal coding agent that points at any OpenAI-compatible or local endpoint