A CONCEPT · PROJECTED FROM THE VINDEX KNOWLEDGE GRAPH
LINEAR ATTENTION
the past folded into state
An attention family that carries the past as a fixed-size recurrent state instead of a growing cache — constant cost per token, however long the context.
Carried as a first-class execution surface — key_heads · key_head_dim · value_heads · value_head_dim · conv_kernel · state_dtype — present only when a model uses it, never inferred from a model name. What changes is not the container's discipline but the memory the layer carries forward.
IN THE GRAPH
linear attention— sibling of →attention
linear attention— belongs to →token mixer
linear attention— carries →recurrent state
linear attention— specialised by →gated deltanet
linear attention— specialised by →kda — kimi delta attention