A CONCEPT · PROJECTED FROM THE VINDEX KNOWLEDGE GRAPH
UP — up_proj
expands hidden representation dimensionality
Expands the hidden state into a much wider intermediate space — more room to transform information than the stream itself allows.
In the worked example, 2,048 numbers become 6,144. The width is temporary: after the gate's judgement applies, the down projection returns the result to the stream's width.
IN THE GRAPH
up — up_proj— paired with →gate — gate_proj
up — up_proj— belongs to →feed-forward
up — up_proj— represented by →gate_up — the paired bank