# deltafin

Deltafin runs [[kimi-k3]] — 2.8T parameters, about 1.56 TB of MXFP4 weights — on a single Apple Silicon Mac. It works because a [[mixture-of-experts]] model only *touches* a small slice of itself per token, so the rest never has to be in RAM. What it produces is one token every 14.6 seconds on the maintainer's M1 Max, which the README is refreshingly blunt about: "a research artifact, not a practical chat setup."

## How it splits the model

Three pieces, each handled differently.

The **resident spine** — attention, shared experts, latent projections, embeddings — is about 114 GB in bf16 or 53–60 GB quantized to int8. It is downloaded once and read layer by layer from local storage on every token, then computed on MPS, CUDA, or CPU.

The **82,432 routed experts** are about 1.45 TB. K3's router picks 16 per layer across 92 layers, so a token reads 25.8 GB of expert data and nothing else. Install them all locally if you have the disk, or let Deltafin fetch each missing one from Hugging Face with a single HTTP range request into a growing disk cache.

The **forward pass** is Moonshot's own modeling code, unmodified, with a pure-PyTorch shim standing in for the CUDA-only `fla` kernels it expects. That shim reimplements [[kimi-delta-attention]]'s recurrence, short convolution, and gated norm; chunked and step-by-step execution agree to about 1e-9. At decode the recurrence runs on CPU, because KDA's small state fits better there than in a series of GPU dispatches.

Deltafin's implementation notes are the clearest public account of K3's shape: 93 decoder layers, 69 KDA and 24 MLA, 896 experts per layer, top-16 routing, 96 safetensors shards.

## The two install modes

```
./venv/bin/python tools/setup_k3.py --full     # ~1.7 TB, 5-10 hours, resumable
./venv/bin/python tools/setup_k3.py --stream   # ~215 GB, ~30 minutes
```

One number decides between them. Those 25.8 GB of expert data per token take about 4 seconds off local disk and minutes over the network, so a full install decodes at 14.6 s/token and a streaming install at roughly 3+ minutes per token for anything not already cached. Streaming is a way to try it without committing 1.7 TB; `tools/fetch_experts_all.py` finishes the job later, resumable and partial-range-capable, with no reinstall. The server and CLI both print a startup warning while still in streaming mode.

Usage is a CLI or an OpenAI-compatible server:

```
./venv/bin/python tools/kimi_run.py --chat --prompt "What are the three largest moons of Saturn?"
./venv/bin/python tools/serve_openai.py --port 8000
```

`/v1/chat/completions`, `/v1/completions`, `/v1/models` and streaming all work, so anything reading `OPENAI_BASE_URL` can point at it — the same swap [[cc-mirror]] automates for Claude Code variants. The caveats are the interesting part: set client timeouts to hours, decoding is greedy only (`temperature` and `top_p` are accepted and ignored), one request at a time with a 429 for the second, and coding agents are "a curiosity, not a workflow" because their long system prompts make prefill expensive.

## Where the time goes

The maintainer's reference machine is a first-generation M1 Max, 10-core CPU, 32-core GPU, 64 GB, internal NVMe, with the full model local, int8 spine and output head, Metal MoE, greedy decoding, tracing off. Six exact full-model runs from balanced ABBA/BAAB campaigns, medians reported:

| Metric | First working version | Current |
| --- | --- | --- |
| Prefill / first token (5-token prompt) | 2,429 s | 28.0 s (24.9–37.9) |
| Steady decode, experts local | ~20 min/token | 14.6 s/token (0.0503–0.0779 tok/s) |
| Decode, experts streamed | ~20 min/token | ~3 min/token, network-bound |

That is roughly 4.1 tokens per minute, and the per-token budget breaks down as ~5 s waiting on the resident spine read, ~4.3 s reading the 16 selected experts per layer, ~3 s applying the spine, ~2 s attention and norms across 93 layers, ~1 s of MoE matmuls. Decode is bound by disk bandwidth on the spine: 53 GB re-read every token at the ~7 GB/s this access pattern sustains is about 7.5 s of the 14.6 s median, and the only cures are more RAM or a smaller spine.

## The techniques that bought the 82×

Each was measured on real weights before being kept, and most are adaptations rather than inventions — the README credits colibri, antirez's ds4, and llama.cpp/ggml explicitly.

Expert fetches are **coalesced**: all six of an expert's tensors turned out to be contiguous in the shard files (they checked all 82,432), so one expert is a single 17.55 MB range request over keep-alive connections, about 6.4× faster than per-tensor fetching. The disk cache stores shard bytes verbatim — no container format, no parsing. A layer's 16 experts are read together by a thread pool using `pread` instead of being demand-faulted page by page; on macOS, `F_NOCACHE` keeps 25 GB/token of expert traffic from evicting the page cache the spine needs, and the cold-read comparison was 0.87 GB/s faulting versus 6.85 GB/s reading, worth 40 s → 4.3 s per token.

On the compute side, a **fused MXFP4 dequant+GEMV** kernel (`tools/fused_gemv.c`) dequantizes and multiplies in one pass via a 16-entry table lookup, applying the e8m0 scale as integer arithmetic on the fp32 exponent. NEON on aarch64, SSSE3/FMA on x86-64, bit-for-bit against the reference. A custom Metal dequant kernel replaced a row-broadcast multiply that MPS ran at 43 GB/s (against 334 GB/s for a plain copy of the same bytes) with a fused int8→fp32 + scale + copy at 297 GB/s, taking per-layer load from 118 ms to 21 ms with `max|diff| = 0`. Because all 69 KDA layers share one set of tensor shapes and all 24 MLA layers another, two persistent device-resident "template" layers receive each layer's weights via `copy_()`, avoiding allocator churn.

Quantization here is an I/O decision more than an arithmetic one — see [[llm-quantization]] for the general framing. The int8 spine halves per-token resident I/O; in their checks the top-5 next-token candidates kept their order and the top logit moved by 0.07%. A packed MPS int8 output head gave +17.3% steady decode, +23.1% prefill, +26.8% wall throughput, and cut resident head storage from 4.7 GB to 1.17 GB. The packed int8 KDA Q/K/V path is still an opt-in (`K3_INT8_KDA_QKV=1`) because its first A/B measured +2.8% decode and −4.7% prefill with overlapping run ranges.

N-gram [[speculative-decoding|speculation]] is on by default and lossless: drafts come free from suffix matching against the generated text, get verified in a two-position batch, and a rejected draft restores state bit-for-bit by retaining the old immutable state objects rather than cloning ~475 MB. It pays off here for an unusual reason — resident I/O and compute, not expert fetching, dominate a warm token, so the second position rides along nearly free.

## Platforms and their asymmetries

Apple Silicon macOS gets MPS plus a Metal MoE kernel with native CPU fallback. NVIDIA Linux is deliberately described as hybrid: CUDA accelerates the resident spine and attention, but routed MXFP4 experts still run in the native CPU kernel because no CUDA MXFP4 MoE kernel exists yet. Linux x86-64 needs the x86-64-v3 level (AVX2, FMA3) and uses an SSSE3/FMA kernel; aarch64 uses NEON. `build_native.py` picks `.dylib` or `.so`, applies host ISA flags, and validates symbols and ABI before installing anything.

A community run on an NVIDIA DGX Spark (GB10 Grace-Blackwell, 128 GB unified LPDDR5X) produced the most instructive number in the README. Across four configurations, int8 + CUDA finished in 221 s against 865 s for bf16 + CPU — roughly 4× end to end, not the 2× that halving spine bytes would suggest. The 107 GB bf16 spine left no room for the expert page cache, while the ~53 GB int8 spine freed enough headroom that preload wait collapsed from 676 s to 18 s. Quantization changed the I/O regime, not just the arithmetic. That is the same lever [[moe-cpu-offload]] pulls in llama.cpp, one storage tier further down.

## Limitations

The maintainer states them plainly. A 14.6-second median token is nowhere near interactive; long prompts are expensive because prefill touches many experts; there is no quality harness yet, so lossy speed/quality trade-offs are argued rather than measured (average NLL against the official API is on the roadmap, borrowed from ds4). Every headline number comes from one M1 Max, and the DGX Spark figures are a single community run in pull request #2, not maintainer-replicated. The roadmap's first item is a native CUDA MXFP4 MoE kernel.

Output is greedy and reproducible — the same prompt yields the same tokens run after run — which is what makes the A/B measurements above meaningful in the first place.

Tracked on [[toolbox/watchlist]]: worth re-checking once the CUDA MXFP4 kernel and the NLL quality harness land, since both decide whether this stays an existence proof or becomes a usable way to run K3 outside a datacenter. In the meantime it is the counterweight to [[local-ai-is-not-opus]]: you *can* run the frontier open model at home, and the price is measured in seconds per token.

Deltafin's own code is [MIT](https://github.com/gavamedia/deltafin); `tools/fla/` is a port of semantics from flash-linear-attention (MIT), and K3's weights and modeling code stay under Moonshot's license and are downloaded at setup rather than vendored. The project has no affiliation with Moonshot AI.