I Want Local Models to Work

Armin Ronacher (Flask, Sentry, Pi) argues that the local-LLM experience is worse than it has any right to be — and the failure isn't model quality, it's product polish across the stack. The post is part position paper, part announcement of his pi-ds4 extension that wraps Salvatore Sanfilippo's ds4.c inside Pi.

The hosted-vs-local UX gap is the framing. Pasting an API key into Pi is "boring" in the good sense — provider, key, done. Doing the same locally means picking an inference engine, then a model, then a quantization, then a chat template, then a context size, then debugging JSON config across multiple layers, then discovering one of those choices quietly degraded the model. None of this is necessary; all of it is currently the user's problem.

Three concrete failure modes:

Tool parameter streaming isn't implemented in most local stacks. The OpenAI-style completions API supports streaming the tool-call arguments incrementally, the way text tokens stream. Most local servers buffer the entire tool call and emit it atomically. Consequences: dead-connection-vs-slow-model is indistinguishable, you can't see what bash invocation is being assembled, you can't interrupt early. This is invisible to anyone benchmarking model quality but ruinous for coding-agent UX.

Stack fragmentation. llama.cpp, Ollama, LM Studio, MLX, Transformers, vLLM — all great projects individually, but for any given model the actual runtime behavior depends on a long chain of small decisions (chat template rendering, reasoning-token handling, tool-call format translation, context-window honesty, KV cache wiring, quantization match to hardware) that most users won't trace. Result: people try a local model, get a mediocre experience, dismiss the model. "Big old Python packaging vibes."

No critical mass on any one combination. New models drop weekly; attention moves to the next thing instead of polishing a single (model, hardware, engine, harness) tuple end-to-end. Hosted providers don't ship "a bag of weights and a config" — they ship a product. Local needs to start doing the same.

The DS4 bet: ds4.c is antirez's deliberately narrow inference engine — DeepSeek V4 Flash on Macs with 128GB+, Metal path, model-specific loading and prompt rendering, KV handling with SSD spillover, server API. Not a generic GGUF runner. Ronacher's pi-ds4 registers this as a first-class Pi provider: compiles and starts ds4-server on demand, downloads/builds runtime, picks quantization automatically, lifecycle-managed. No knobs intentionally, because the goal is to figure out how to set the knobs automatically.

The frame: this isn't about hiding complexity, it's about putting it in one place where it can be improved. Pick one winner, polish it, then scale to more configs.

Why it matters: connects to hold-on-to-your-hardware (Mac Studio with that much RAM is currently unobtainable), titit-local-ai (Neward's similar argument), deepseek (V4 Flash as the model that's actually plausible to run locally if you have the box), and ai-subsidy-economics (Ronacher's "hammer locked behind a subscription in another country" line is the same critique).

Cross-references: lucumr-blog, titit-local-ai, deepseek, hold-on-to-your-hardware, deepseek-v4-roleplay-instruct, local-llm-16gb-vram-tests (a worked example of exactly the kind of operator expertise Ronacher says shouldn't be required — --n-cpu-moe, triple thinking-blockage, KV-cache precision swap), steering-vectors-interesting-again (the dir-steering/ directory in ds4 is the case Goedecke uses for why steering is suddenly practical), llama-cpp, opencode.