#attention

Wiki 2

• Kimi Delta Attention (KDA) DeltaNet linear attention derived from softmax attention up to KDA's per-key-channel forget gate
• Kimi K3 Raschka on Moonshot's 2.8T open-weight K3 — LatentMoE, KDA, attention residuals, NoPE everywhere