A CONCEPT · PROJECTED FROM THE VINDEX KNOWLEDGE GRAPH
MLA — MULTI-HEAD LATENT ATTENTION
the cache compressed through latents
An attention family that stores its key/value past through a low-rank latent bottleneck — full-attention memory at a fraction of the cache bytes.
Its surface declares num_heads · kv_lora_rank · qk_nope_head_dim · qk_rope_head_dim · v_head_dim: the rope and no-rope halves of the key are separate declared facts, and the latent A/B projections with their norm are operands like any others. It keeps true KV state — compressed, not replaced.
IN THE GRAPH
mla — multi-head latent attention— sibling of →attention
mla — multi-head latent attention— sibling of →kda — kimi delta attention
mla — multi-head latent attention— distinguished by →kv state vs recurrent state