A CONCEPT · PROJECTED FROM THE VINDEX KNOWLEDGE GRAPH
EXPERT
another gate-up-down, kept many times
In the gated-MLP form, nothing exotic: one more feed-forward triple — gate, up, down — kept dozens of times so a router can choose per token. Other expert programmes exist; the manifest names which one a bank runs.
A mixture-of-experts layer grows many ordinary feed-forward networks instead of one enormous one. Most experts stay dark on any given token, which is why residency policy exists at all.
IN THE GRAPH
expert— is a →feed-forward
expert— selected by →router