Kimi Delta Attention (KDA)

Kimi Delta Attention is the linear-attention variant used in kimi-k3 and Kimi Linear. Written out cold, its state update looks arbitrary:

$$ \widetilde S_t = S_{t-1}\operatorname{Diag}(\alpha_t), \quad \hat v_t = \widetilde S_t k_t, \quad e_t = \beta_t(v_t - \hat v_t), \quad S_t = \widetilde S_t + e_t k_t^{\mathsf T}, \quad o_t = S_t,d_k^{-1/2} q_t $$

Jamie Dborin's derivation at Doubleword rebuilds it one demand at a time, along the chain softmax attention → linear attention → DeltaNet → Gated DeltaNet → KDA. Each step is a single complaint about the previous one. The original uses bra-ket notation; the equations here are in conventional vector form, with $S$ mapping key space to value space, so $S$ is $d_v \times d_k$ and $k_t$ is a unit vector.

Dropping the softmax turns attention into a state

Causal softmax attention computes $T^2$ key-query scores for a length-$T$ sequence. Caching keys and values avoids recomputation but the cache still grows with the sequence, and every new query still has to look at all of it.

The obstacle to rearranging the computation is the softmax denominator, which depends jointly on the current query and every earlier key. Remove it and absorb the $d_k^{-1/2}$ scale into the query, and attention is

$$ o_t = \sum_{i \le t} (k_i^{\mathsf T} q_t), v_i = \Big(\sum_{i \le t} v_i k_i^{\mathsf T}\Big) q_t $$

The whole trick is that $(v k^{\mathsf T})q = (k^{\mathsf T}q),v$: the outer product is a matrix, the inner product is a scalar. Everything about the past collapses into one fixed-size matrix, and attention becomes a write followed by a read:

$$ S_t = S_{t-1} + v_t k_t^{\mathsf T}, \qquad o_t = S_t q_t $$

Cost is now linear in sequence length, and — the reason this matters for a 1M-token context — memory is constant instead of growing per token the way a KV cache does. What was paid for it is softmax's normalization and selectivity.

Addition is not assignment

The linear-attention write is broken in a specific way. Write $v_t k_t^{\mathsf T}$, then immediately query the new state with the same key:

$$ S_t k_t = S_{t-1}k_t + v_t\underbrace{(k_t^{\mathsf T}k_t)}{1} = S{t-1}k_t + v_t $$

The memory does not return $v_t$. It returns $v_t$ plus whatever it already returned. If the old state was already correct, the new state now returns double. And since keys are not mutually orthogonal, every write leaks into every earlier association. The update behaves like += where the intent was =.

DeltaNet: write the error instead of the value

DeltaNet fixes the write. Before storing token $t$, ask the memory what it currently associates with the new key, $\hat v_t = S_{t-1}k_t$, and add only the difference, scaled by a learned write strength $\beta_t \in [0,1]$:

$$ e_t = \beta_t(v_t - S_{t-1}k_t), \qquad S_t = S_{t-1} + e_t k_t^{\mathsf T} $$

Read the same key back and

$$ S_t k_t = (1-\beta_t)S_{t-1}k_t + \beta_t v_t $$

At $\beta_t = 1$ the read returns exactly $v_t$; smaller values interpolate. The correction is also local: for any $x$ orthogonal to $k_t$, $(S_t - S_{t-1})x = e_t(k_t^{\mathsf T}x) = 0$, so the rank-one write changes the response along the selected key direction and leaves every orthogonal direction untouched.

The same update falls out of a completely different framing. Treat the current pair as one training example for the linear map $S$ under a reconstruction loss $\mathcal L_t(S) = \tfrac12\lVert Sk_t - v_t\rVert_2^2$. Its gradient is $(Sk_t - v_t)k_t^{\mathsf T}$ — visibly an outer product of a value-space error with the key at which the error was observed. One gradient-descent step of size $\beta_t$ from $S_{t-1}$ reproduces the delta rule exactly. So $\beta_t$ is a replacement strength if you read the state as a memory, a learning rate if you read it as online learning, and the change is a rank-one update either way.

Expanding gives DeltaNet as a structured state transition:

$$ S_t = S_{t-1}\big(I - \beta_t k_t k_t^{\mathsf T}\big) + \beta_t v_t k_t^{\mathsf T} $$

For a unit key, $I - \beta_t k_t k_t^{\mathsf T}$ has eigenvalue $1-\beta_t$ along the current key and $1$ everywhere else. It erases the old association at that key before writing the new one.

Gated DeltaNet: sometimes everything should fade

DeltaNet fixes the write but not the lifetime of what was written. Once a token is folded into $S_t$, a read $S_t q = \sum_{i\le t}(k_i^{\mathsf T}q)v_i$ cannot skip it. The delta rule cleans up around the current key; stale content in other directions still contributes to every future read.

Gated DeltaNet adds a learned scalar retention gate $\alpha_t \in [0,1]$ applied before the delta rule runs:

$$ \widetilde S_t = \alpha_t S_{t-1} \ (\text{forget}), \quad \hat v_t = \widetilde S_t k_t \ (\text{predict}), \quad e_t = \beta_t(v_t - \hat v_t) \ (\text{correct}), \quad S_t = \widetilde S_t + e_t k_t^{\mathsf T} \ (\text{write}) $$

The order is load-bearing. Predicting before forgetting would compute the error against a memory that is not the one being updated. The two mechanisms solve different problems: the delta rule does targeted replacement, the gate does global erasure.

What remains wrong is that $\alpha_t$ is one number for the entire matrix. Every key channel must be retained or dropped at the same rate.

KDA: one gate per key channel

KDA promotes the scalar to a vector $\alpha_t \in [0,1]^{d_k}$ and puts it on a diagonal, $D_t = \operatorname{Diag}(\alpha_t)$. Because the state maps keys to values, the key channels are the columns of $S$, so right-multiplication decays each one independently:

$$ \widetilde S_t = S_{t-1}D_t $$

Everything after that is the delta rule already derived. Conceptually the entire change from Gated DeltaNet is $\alpha_t \longrightarrow \operatorname{Diag}(\alpha_t)$, and the effect is that one channel can be cleared while another is held.

Expanding the recurrence shows what kind of transition this is:

$$ S_t = S_{t-1}\underbrace{D_t\big(I - \beta_t k_t k_t^{\mathsf T}\big)}_{A_t} + \beta_t v_t k_t^{\mathsf T}, \qquad A_t = D_t - (D_t k_t)(\beta_t k_t)^{\mathsf T} $$

$A_t$ is a diagonal matrix minus a rank-one matrix — diagonal-plus-low-rank, DPLR. Worth keeping the shapes straight: DPLR describes the $d_k \times d_k$ transition acting on key space, while the memory itself is still the $d_v \times d_k$ matrix $S_t$.

The four-step family, compressed:

Implementations store $g_t = \log\alpha_t \le 0$ and recover the retention factors as $\exp(g_t)$. In the transposed $d_k \times d_v$ layout the reference code uses, the whole recurrence is five lines:

state = state * g_t.exp().unsqueeze(-1)
prediction = einsum("bhkv,bhk->bhv", state, k_t)
residual = beta_t.unsqueeze(-1) * (v_t - prediction)
state = state + einsum("bhk,bhv->bhkv", k_t, residual)
output = einsum("bhk,bhkv->bhv", q_t * scale, state)

Two schedules for the same recurrence

The article's second half is about execution, and its point is that KDA's recurrent and chunkwise kernels are not two mechanisms — they are two schedules producing identical states and outputs.

Fused recurrent is the literal transcription, one Triton program per sequence, value head, and 32-wide value tile (BV = 32, BK = next_power_of_2(K)), each owning a [BK, BV] tile of the state and looping over tokens in order. The prediction and read are reductions, the write is an outer product. That suits decode, where only one token exists at a time, and suits it badly for training, because vector operations never become the large matrix multiplications tensor cores want.

Chunkwise processes $C$ tokens at once and has to reproduce the same result despite each token's error depending on writes made earlier in the same chunk. The resolution is to compute provisional errors first, pretending no in-chunk writes exist,

$$ \bar e_i = \beta_i\big(v_i - S_c D_{0:i}k_i\big) $$

and then correct them. Every correction term is a scalar coefficient times an earlier error, so define

$$ \rho_{ij} = \beta_i, k_j^{\mathsf T} D_{j+1:i}, k_i, \qquad j < i $$

where $D_{j+1:i}$ is the cumulative decay carrying a write at $j$ forward to $i$. Collecting the $\rho_{ij}$ into a strictly lower-triangular $R_c$ makes the whole dependency chain one linear system, $E_c = \bar E_c\big((I + R_c)^{-1}\big)^{\mathsf T}$. No dense inverse is needed: $I + R_c$ is triangular with unit diagonal, so this is a causal triangular solve applied per value channel.

With the true errors in hand, the chunk's two required results are both matrix products. The state fast-forwards in one shot,

$$ S_{c+1} = S_c D_{0:C-1} + E_c K_c^{\mathrm{end}} $$

where $K_c^{\mathrm{end}}$ holds each key decayed to the chunk boundary. The outputs come from a second lower-triangular coefficient matrix $A^{qk}c$ built from $\chi{ij} = s,k_j^{\mathsf T}D_{j+1:i}q_i$ for $j \le i$ — the diagonal is included because KDA reads after writing, so token $i$ sees its own write:

$$ O_c = s,S_c Q_c^{\mathrm{boundary}} + E_c\big(A^{qk}_c\big)^{\mathsf T} $$

The first term reads the decayed incoming state, the second adds the causal contribution of in-chunk writes.

The real Triton implementation is a pipeline of launches rather than one kernel. It computes chunk-local cumulative log decays first, so $D_{j+1:i}$ comes from a difference of two prefix sums instead of multiplying a chain of retention vectors; builds the $A^{qk}$ and $A^{kk}$ interaction matrices, with $A^{kk}$ feeding a WY-style representation of the chunk's corrected writes; runs one state kernel that performs the only inter-chunk scan; and then lets an output kernel compute all chunks and tiles in parallel. The shipped source fuses further, computing 16-token diagonal interaction blocks before a combined off-diagonal and triangular-solve kernel.

Why this shape is useful outside a datacenter

KDA's recurrent state is small and fixed, which is what makes long context on modest hardware tractable at all. deltafin runs the KDA layers of K3 on CPU at decode time precisely because the state fits better there than in a chain of GPU dispatches, and its pure-PyTorch shim for the CUDA-only fla kernels agrees with the chunked path to about 1e-9. Moonshot's own credit for feasibility on a laptop goes to KDA rather than to any of the quantization work.

Qwen took the Gated DeltaNet step of the family instead: qwen3-8-flash-next builds each of its 12 blocks from three Gated DeltaNet layers and one full-attention layer.