deltafin

Deltafin runs kimi-k3 — 2.8T parameters, about 1.56 TB of MXFP4 weights — on a single Apple Silicon Mac. It works because a mixture-of-experts model only touches a small slice of itself per token, so the rest never has to be in RAM. What it produces is one token every 14.6 seconds on the maintainer's M1 Max, which the README is refreshingly blunt about: "a research artifact, not a practical chat setup."

How it splits the model

Three pieces, each handled differently.

The resident spine — attention, shared experts, latent projections, embeddings — is about 114 GB in bf16 or 53–60 GB quantized to int8. It is downloaded once and read layer by layer from local storage on every token, then computed on MPS, CUDA, or CPU.

The 82,432 routed experts are about 1.45 TB. K3's router picks 16 per layer across 92 layers, so a token reads 25.8 GB of expert data and nothing else. Install them all locally if you have the disk, or let Deltafin fetch each missing one from Hugging Face with a single HTTP range request into a growing disk cache.

The forward pass is Moonshot's own modeling code, unmodified, with a pure-PyTorch shim standing in for the CUDA-only fla kernels it expects. That shim reimplements kimi-delta-attention's recurrence, short convolution, and gated norm; chunked and step-by-step execution agree to about 1e-9. At decode the recurrence runs on CPU, because KDA's small state fits better there than in a series of GPU dispatches.

Deltafin's implementation notes are the clearest public account of K3's shape: 93 decoder layers, 69 KDA and 24 MLA, 896 experts per layer, top-16 routing, 96 safetensors shards.

The two install modes

./venv/bin/python tools/setup_k3.py --full     # ~1.7 TB, 5-10 hours, resumable
./venv/bin/python tools/setup_k3.py --stream   # ~215 GB, ~30 minutes

One number decides between them. Those 25.8 GB of expert data per token take about 4 seconds off local disk and minutes over the network, so a full install decodes at 14.6 s/token and a streaming install at roughly 3+ minutes per token for anything not already cached. Streaming is a way to try it without committing 1.7 TB; tools/fetch_experts_all.py finishes the job later, resumable and partial-range-capable, with no reinstall. The server and CLI both print a startup warning while still in streaming mode.

Usage is a CLI or an OpenAI-compatible server:

./venv/bin/python tools/kimi_run.py --chat --prompt "What are the three largest moons of Saturn?"
./venv/bin/python tools/serve_openai.py --port 8000

/v1/chat/completions, /v1/completions, /v1/models and streaming all work, so anything reading OPENAI_BASE_URL can point at it — the same swap cc-mirror automates for Claude Code variants. The caveats are the interesting part: set client timeouts to hours, decoding is greedy only (temperature and top_p are accepted and ignored), one request at a time with a 429 for the second, and coding agents are "a curiosity, not a workflow" because their long system prompts make prefill expensive.

Where the time goes

The maintainer's reference machine is a first-generation M1 Max, 10-core CPU, 32-core GPU, 64 GB, internal NVMe, with the full model local, int8 spine and output head, Metal MoE, greedy decoding, tracing off. Six exact full-model runs from balanced ABBA/BAAB campaigns, medians reported:

That is roughly 4.1 tokens per minute, and the per-token budget breaks down as ~5 s waiting on the resident spine read, ~4.3 s reading the 16 selected experts per layer, ~3 s applying the spine, ~2 s attention and norms across 93 layers, ~1 s of MoE matmuls. Decode is bound by disk bandwidth on the spine: 53 GB re-read every token at the ~7 GB/s this access pattern sustains is about 7.5 s of the 14.6 s median, and the only cures are more RAM or a smaller spine.

The techniques that bought the 82×

Each was measured on real weights before being kept, and most are adaptations rather than inventions — the README credits colibri, antirez's ds4, and llama.cpp/ggml explicitly.

Expert fetches are coalesced: all six of an expert's tensors turned out to be contiguous in the shard files (they checked all 82,432), so one expert is a single 17.55 MB range request over keep-alive connections, about 6.4× faster than per-tensor fetching. The disk cache stores shard bytes verbatim — no container format, no parsing. A layer's 16 experts are read together by a thread pool using pread instead of being demand-faulted page by page; on macOS, F_NOCACHE keeps 25 GB/token of expert traffic from evicting the page cache the spine needs, and the cold-read comparison was 0.87 GB/s faulting versus 6.85 GB/s reading, worth 40 s → 4.3 s per token.

On the compute side, a fused MXFP4 dequant+GEMV kernel (tools/fused_gemv.c) dequantizes and multiplies in one pass via a 16-entry table lookup, applying the e8m0 scale as integer arithmetic on the fp32 exponent. NEON on aarch64, SSSE3/FMA on x86-64, bit-for-bit against the reference. A custom Metal dequant kernel replaced a row-broadcast multiply that MPS ran at 43 GB/s (against 334 GB/s for a plain copy of the same bytes) with a fused int8→fp32 + scale + copy at 297 GB/s, taking per-layer load from 118 ms to 21 ms with max|diff| = 0. Because all 69 KDA layers share one set of tensor shapes and all 24 MLA layers another, two persistent device-resident "template" layers receive each layer's weights via copy_(), avoiding allocator churn.

Quantization here is an I/O decision more than an arithmetic one — see llm-quantization for the general framing. The int8 spine halves per-token resident I/O; in their checks the top-5 next-token candidates kept their order and the top logit moved by 0.07%. A packed MPS int8 output head gave +17.3% steady decode, +23.1% prefill, +26.8% wall throughput, and cut resident head storage from 4.7 GB to 1.17 GB. The packed int8 KDA Q/K/V path is still an opt-in (K3_INT8_KDA_QKV=1) because its first A/B measured +2.8% decode and −4.7% prefill with overlapping run ranges.

N-gram speculation is on by default and lossless: drafts come free from suffix matching against the generated text, get verified in a two-position batch, and a rejected draft restores state bit-for-bit by retaining the old immutable state objects rather than cloning ~475 MB. It pays off here for an unusual reason — resident I/O and compute, not expert fetching, dominate a warm token, so the second position rides along nearly free.

Platforms and their asymmetries

Apple Silicon macOS gets MPS plus a Metal MoE kernel with native CPU fallback. NVIDIA Linux is deliberately described as hybrid: CUDA accelerates the resident spine and attention, but routed MXFP4 experts still run in the native CPU kernel because no CUDA MXFP4 MoE kernel exists yet. Linux x86-64 needs the x86-64-v3 level (AVX2, FMA3) and uses an SSSE3/FMA kernel; aarch64 uses NEON. build_native.py picks .dylib or .so, applies host ISA flags, and validates symbols and ABI before installing anything.

A community run on an NVIDIA DGX Spark (GB10 Grace-Blackwell, 128 GB unified LPDDR5X) produced the most instructive number in the README. Across four configurations, int8 + CUDA finished in 221 s against 865 s for bf16 + CPU — roughly 4× end to end, not the 2× that halving spine bytes would suggest. The 107 GB bf16 spine left no room for the expert page cache, while the ~53 GB int8 spine freed enough headroom that preload wait collapsed from 676 s to 18 s. Quantization changed the I/O regime, not just the arithmetic. That is the same lever moe-cpu-offload pulls in llama.cpp, one storage tier further down.

Limitations

The maintainer states them plainly. A 14.6-second median token is nowhere near interactive; long prompts are expensive because prefill touches many experts; there is no quality harness yet, so lossy speed/quality trade-offs are argued rather than measured (average NLL against the official API is on the roadmap, borrowed from ds4). Every headline number comes from one M1 Max, and the DGX Spark figures are a single community run in pull request #2, not maintainer-replicated. The roadmap's first item is a native CUDA MXFP4 MoE kernel.

Output is greedy and reproducible — the same prompt yields the same tokens run after run — which is what makes the A/B measurements above meaningful in the first place.

Tracked on watchlist: worth re-checking once the CUDA MXFP4 kernel and the NLL quality harness land, since both decide whether this stays an existence proof or becomes a usable way to run K3 outside a datacenter. In the meantime it is the counterweight to local-ai-is-not-opus: you can run the frontier open model at home, and the price is measured in seconds per token.

Deltafin's own code is MIT; tools/fla/ is a port of semantics from flash-linear-attention (MIT), and K3's weights and modeling code stay under Moonshot's license and are downloaded at setup rather than vendored. The project has no affiliation with Moonshot AI.