A CONCEPT · PROJECTED FROM THE VINDEX KNOWLEDGE GRAPH
ATTENTION
each token reads the past
The first move of a layer: every token asks questions of the tokens before it, and information flows back where the questions match.
Attention gives each token three faces made by three tensors — a query, a key, a value — plus an output projection that writes the gathered result back to the stream.
IN THE GRAPH
attention— contains →q — q_proj
attention— contains →k — k_proj
attention— contains →v — v_proj
attention— contains →o — o_proj
attention— belongs to →layer
attention— belongs to →token mixer
attention— specialised by →gated attention