THE ANATOMY · BETWEEN THE PHYSICS AND THE CONTAINER
WHAT A MODEL ACTUALLY CONTAINS
The addresses in this exhibition — q_proj, gate_proj, router, expert — name real machinery. This chapter is that machinery, opened, in one worked, conventional decoder shape — a pure-SSM stack, like the Mamba2 witness on the Record, carries none of the attention below: hidden 2,048 · intermediate 6,144 · 24 layers · 32 experts, top-4. No mathematics required; ten minutes, and the rest of the site reads in plain sight.
layer.17.mlp.down_proj reads like a filename — until you know the machine it names.
WHAT THIS FIXES
The Physics explained why the bytes are heavy. The Container will explain how VINDEX3 holds them. Between the two sits a vocabulary cliff: gate, up, down, query, key, value, router, expert. A glossary would be documentation debt. Instead: the machine itself, one part at a time, each part ending at the address you will meet again downstairs.
A conventional decoder layer is built around two main transformations: attention looks backwards along the sentence, and the feed-forward network transforms what it found. (Other attention families — linear, KDA, MLA — slot into the same seat; the Execution chapter carries them.) Each result is added to a running stream rather than replacing it — which is why a layer can be measured, attributed, or skipped without the story falling apart.
ATTENTION
The first thing a layer does is look backwards. For that, every token is given three faces, each made by its own tensor: a query — what am I looking for; a key — what do I contain; a value — what do I return. Queries meet keys, and where they agree, values flow. Hold each part below.
What are Q, K and V in a transformer?
Query, key and value are three projections of the same token, each made by its own weight tensor. The query says what a token is looking for; the key says what it contains; the value is what it hands over. Queries are compared against every earlier key, and wherever they agree, that token's value flows forward — that comparison is attention.
one sentence · the stream holds a hidden state per token
“it” builds its question — which earlier thing do I stand for?
QUERY — what am I looking for?
Turns this token's state into the questions it asks of every token before it. A verb asking for its subject; a pronoun asking who it stands for. The query is compared against every earlier token's key, and where they agree strongly, that token's value flows back.
layer.17.attention.q_proj — readable now
THE FEED-FORWARD NETWORK
The second thing a layer does is transform what attention gathered — and this is where most of a model's weight lives. The shape of the move is almost physical: make the space bigger, decide what gets through, bring it back home. Step through it.
What do gate_proj, up_proj and down_proj do?
In a gated MLP, up_proj widens a token's representation into a larger space, gate_proj decides — value by value — how much of that widened signal passes, and down_proj brings the result back to the model's hidden size before it rejoins the residual stream. Most of a model's weight lives in these three tensors, repeated per layer.
2,048 values
The stream arrives.
2,048 numbers — the token as the layers so far understand it.
Expand, judge, compress, add back. Three tensors — gate, up, down — and that is the entire feed-forward network. Everything a layer knows lives in how these grids of numbers steer the multiplication.
In the gated-MLP form shown here, an expert is not exotic machinery. It is the same gate, up, and down — kept thirty-two times, chosen four at a time.
MIXTURE OF EXPERTS
When models grew, they did not grow one enormous feed-forward network. They grew many ordinary ones — experts — and added a router: a small tensor that reads each token and picks which few experts answer it. Route a token below and watch most of the model stay dark.
What is a Mixture-of-Experts router?
An MoE model keeps many ordinary feed-forward networks — experts — and a router: a small tensor that reads each token and scores every expert, activating only the top few. Most of the model stays dark on any given token, which is how parameter count grows without the per-token compute growing with it. The router's scores decide everything, so its precision matters more than its size.
Each small square is an expert — and in the gated-MLP form shown here, an expert is nothing exotic: another gate–up–down triple, the three bars of its glyph. (Other expert programmes exist; the manifest names which one a bank runs.) A mixture-of-experts layer keeps thirty-two of them and lets a router pick four per token. Which is why the Physics chapter could say the routing is the model's and the residency is yours — now you know what those resident bytes are.
NORM AND RESIDUAL
Two quieter parts complete the layer. Norms keep the numbers in a workable range before each move — thermostats, not thinkers. And the residual stream is the discipline that everything is added to a running representation, never overwritten — the reason a layer's contribution can be measured, attributed, or skipped at all.
A generic explainer stops here: that was a transformer. VINDEX3 keeps going.
ONE ADDRESS, ALL THE WAY DOWN
layer.17.mlp.down_proj
A named part of the system graph: layer seventeen's feed-forward compression, bound to its role — not a filename pattern, a judged fact the container carries as data.
The grid of numbers you just watched bring the wide space back home — its shape and dtype declared, not inferred.
The physically present encodings of that tensor, side by side. A profile selects one; selecting an absent one fails closed before a byte is read.
Where the chosen representation actually lives — its segment, its byte range, its payload hash recorded at encode and re-checked at verify.
A weights file stops at 'this is a down projection.' A container continues: its identity, its representations, its byte range, its provenance, the authority selecting it. That descent is the whole format.
layer.17.mlp.down_proj — semantic component · physical tensor · representation · bytes.
Every definition on this page comes from the VINDEX knowledge graph — the same entries Ask answers from, the same entries the Explorer's DESCRIBE reaches. Three surfaces, one vocabulary.
You now hold the vocabulary the rest of the exhibition assumes. Next: the file that keeps every one of these parts named, present, and checkable — or open a real one and walk it yourself.
PUBLISHED 29 AUG 2026 · REVISED 30 AUG 2026 · VERSION 3.0 Candidate
CITEsite build 43a1b6b · built 2026-09-02
CITE THIS
Web page · 3.0 Candidate
Hay, C. (2026). The Anatomy. In VINDEX3 Specification (Version 3.0 Candidate). VINDEX3. https://vindex3.org/anatomy
A chapter of the specification, citable on its own. Cite the version rather than the site — the canonical reference, and how to use it, live at /cite.