A CONCEPT · PROJECTED FROM THE VINDEX KNOWLEDGE GRAPH
MIXTURE OF EXPERTS
many feed-forwards, chosen per token
The way models grew: not one enormous feed-forward network, but many ordinary ones with a router choosing a handful per token.
Because most of the model sits idle on any one token, the format stores experts as addressable banks — and a profile decides what stays resident, what pages in, and what never loads.