Kimi K3

Moonshot AI released the weights for Kimi K3 on 2026-07-27. Sebastian Raschka's note the next day reads the architecture diagram rather than the leaderboard, and his headline is deflationary in a useful way: K3 is not a new design. It is a production scale-up of the Kimi Linear model Moonshot released in 2025, 48B grown to 2.8T. That makes it by a wide margin the largest open-weight model available, and almost every component in it had already shipped at 1/60th the size.

The one genuinely new part

Compared to Kimi Linear, the only new component is LatentMoE — the same block Nemotron 3 Ultra uses. The idea borrows from multi-head latent attention: down-project the large linear layers into a lower-dimensional latent space instead of computing them at full width. Where MLA compresses the KV projections, LatentMoE compresses the expert projections. Raschka left it out of his architecture figure because the diagram was already crowded.

Everything else is a swap of a component for a cheaper version of itself. Plain mixture-of-experts became LatentMoE, ordinary attention became a mix of gated multi-head latent attention and kimi-delta-attention. Raschka reads this as the same direction Nemotron 3 and DeepSeek V4 are moving in — see deepseek for that lab's line — where the architecture work is mostly about making inference cheaper rather than making the model smarter.

Attention residuals, the one non-efficiency change

The exception is attention residuals, also inherited from Kimi Linear. It is a modification to the residual path, in the same category as DeepSeek V4's mHC (manifold-constrained Hyper-Connections) but working differently: mHC widens the residual path, while attention residuals connect residuals across layers and weight each connection by an attention score, so the contribution of one layer's residual to another is learned rather than fixed.

The technical report's claim is modest and comes with a price tag attached: consistently better validation loss and slightly better downstream performance, at about 4% added training cost and 2% added inference cost. Publishing the overhead alongside the gain is unusual enough to be worth noting, and it is the only component here whose justification is quality rather than throughput.

NoPE everywhere

K3 has no RoPE layers at all. It uses NoPE — no positional embeddings — in every layer, again inherited from Kimi Linear. The prevailing pattern elsewhere had been to keep RoPE in the local layers (sliding-window attention) and use NoPE only in the global ones. A handful of architectures had gone all-NoPE before, but Raschka believes K3 is the first frontier-scale model to do it.

Raschka doesn't say why it works here, but the KDA layers are a plausible reason: a linear-attention layer with a decaying recurrent state already carries positional information implicitly, since the retention gates make recent writes survive better than old ones. Order ends up encoded in how much of each write is left rather than in an explicit rotation of the query and key.

K3 also has native multimodal support, which Kimi Linear did not.

What the model looks like from outside

The 2.8T figure is total parameters, not active ones, and the routing is sparse enough that the model runs on hardware nowhere near able to hold it. deltafin, which runs K3 on a single Apple Silicon Mac, reports the concrete shape: 93 decoder layers, 69 of them KDA and 24 MLA, with routed experts in 92 layers — 896 experts each, 82,432 in total, top-16 selected per token per layer. The MXFP4 weights come to about 1.56 TB, of which the resident non-expert spine is about 114 GB. Every token touches roughly 25.8 GB of expert data.

The published-weights side and the hosted side arrived the same week. telnyx listed K3 on its inference API on 2026-07-28 and advertises a 1M-token context window and vision input; Modal launched managed K3 endpoints on 2026-07-27, which is what prompted the anecdote in rl-finetune-beats-frontier. Cognition's swe-1-7 was RL-trained from the previous generation of the same family, K2.7, which is the pattern that keeps making these Kimi releases matter beyond their own benchmark rows: the weights become other people's base models.

Two things in the vault attach to the release rather than the design. The weights landing set off another round of the open-weights policy argument, which arguments-against-open-source-ai takes apart days later, using the encryption export-control fight as the precedent for what containment achieved last time. And llm-cross-entropy-similarity is a stylometric read of the same model: the notebook opens with the K3-to-Fable-5 pair pinned, which is the closest the source comes to a claim that Moonshot's prose resembles Anthropic's — a hardcoded default view rather than a result the author argues anywhere.

What the release meant outside the architecture is covered from the open-models side of the vault. Nathan Lambert's kimi-k3-open-weights-escalation calls K3 the first frontier-class open-weight model and puts the open-closed gap at three to five months after it, down from six to nine; K3 is the open leader in all three measurements collected on open-closed-model-gap. It landed fifteen days after his six-months-to-live-for-open-models predicted that an open model reaching that level would trigger a US ban or delay on frontier open weights. On distillation it shows up twice: Anthropic's anthropic-threat-report-september-2026 lists Moonshot among the seven Chinese labs it says distilled Claude, and the appendix of stealing-reasoning-traces-from-proprietary-llm-apis finds K3 reacting unusually strongly to Anthropic reasoning traces, which its authors say is not proof.

Raschka's note covers only the architecture. He mentions there are "several other interesting training tidbits" in the technical report that he did not get to.