A CONCEPT · PROJECTED FROM THE VINDEX KNOWLEDGE GRAPH
FEED-FORWARD
expand, judge, compress, add
The second move of a layer, and where most of a model's weight lives: make the space bigger, decide what gets through, bring it back home.
Three tensors — gate, up, down — are the entire mechanism. Up expands the hidden state into a wider working space, gate judges every channel of it, the two multiply, and down compresses the result back into the stream.
IN THE GRAPH
feed-forward— contains →gate — gate_proj
feed-forward— contains →up — up_proj
feed-forward— contains →down — down_proj
feed-forward— belongs to →layer