# I Want Local Models to Work

Armin Ronacher (Flask, Sentry, Pi) argues that the local-LLM experience is worse than it has any right to be — and the failure isn't model quality, it's product polish across the stack. The post is part position paper, part announcement of his `pi-ds4` extension that wraps Salvatore Sanfilippo's `ds4.c` inside Pi.

The hosted-vs-local UX gap is the framing. Pasting an API key into Pi is "boring" in the good sense — provider, key, done. Doing the same locally means picking an inference engine, then a model, then a quantization, then a chat template, then a context size, then debugging JSON config across multiple layers, then discovering one of those choices quietly degraded the model. None of this is necessary; all of it is currently the user's problem.

Three concrete failure modes:

**Tool parameter streaming isn't implemented in most local stacks.** The OpenAI-style completions API supports streaming the tool-call arguments incrementally, the way text tokens stream. Most local servers buffer the entire tool call and emit it atomically. Consequences: dead-connection-vs-slow-model is indistinguishable, you can't see what bash invocation is being assembled, you can't interrupt early. This is invisible to anyone benchmarking model quality but ruinous for coding-agent UX.

**Stack fragmentation.** llama.cpp, Ollama, LM Studio, MLX, Transformers, vLLM — all great projects individually, but for any given model the actual runtime behavior depends on a long chain of small decisions (chat template rendering, reasoning-token handling, tool-call format translation, context-window honesty, KV cache wiring, quantization match to hardware) that most users won't trace. Result: people try a local model, get a mediocre experience, dismiss the model. "Big old Python packaging vibes."

**No critical mass on any one combination.** New models drop weekly; attention moves to the next thing instead of polishing a single (model, hardware, engine, harness) tuple end-to-end. Hosted providers don't ship "a bag of weights and a config" — they ship a product. Local needs to start doing the same.

The DS4 bet: `ds4.c` is antirez's deliberately narrow inference engine — DeepSeek V4 Flash on Macs with 128GB+, Metal path, model-specific loading and prompt rendering, KV handling with SSD spillover, server API. *Not* a generic GGUF runner. Ronacher's `pi-ds4` registers this as a first-class Pi provider: compiles and starts `ds4-server` on demand, downloads/builds runtime, picks quantization automatically, lifecycle-managed. No knobs intentionally, because the goal is to figure out how to set the knobs automatically.

The frame: this isn't about hiding complexity, it's about putting it in one place where it can be improved. Pick one winner, polish it, then scale to more configs.

Why it matters: connects to [[hold-on-to-your-hardware]] (Mac Studio with that much RAM is currently unobtainable), [[titit-local-ai]] (Neward's similar argument), [[deepseek]] (V4 Flash as the model that's actually plausible to run locally if you have the box), and [[ai-subsidy-economics]] (Ronacher's "hammer locked behind a subscription in another country" line is the same critique).

Cross-references: [[lucumr-blog]], [[titit-local-ai]], [[deepseek]], [[hold-on-to-your-hardware]], [[deepseek-v4-roleplay-instruct]], [[local-llm-16gb-vram-tests]] (a worked example of exactly the kind of operator expertise Ronacher says shouldn't be required — `--n-cpu-moe`, triple thinking-blockage, KV-cache precision swap), [[steering-vectors-interesting-again]] (the `dir-steering/` directory in `ds4` is the case Goedecke uses for why steering is suddenly practical), [[toolbox/llama-cpp]], [[toolbox/opencode]].