How Might Language Models Store Facts?

When a language model completes “Michael Jordan plays the sport of” with “basketball,” where is that association stored? Research suggests that multilayer perceptrons (MLPs) play a role, but a complete mechanistic account of factual recall remains an open problem.

I will unpack the MLP computation using a deliberately simplified example. The example shows how a block could store a fact, not a claim that an actual model stores it in exactly this way.

A transformer converts tokens into vectors, then repeatedly applies attention, MLP blocks and normalization. Attention lets positions exchange information; later vectors can combine context with knowledge represented by learned weights.

Directions in a high-dimensional embedding space can represent features. As a familiar illustration, a difference between word embeddings can sometimes capture a relationship shared by other words, though contextual vectors contain much more than a single word’s meaning.

For the toy example, assume distinct directions encode the first name Michael, the surname Jordan and basketball. A dot product indicates whether a vector aligns with each feature; we also assume attention has already combined the two name tokens.

The MLP processes each token’s vector separately and in parallel. It produces another vector of the same size, which is added back to the input as a residual update; the tokens do not exchange information within this step.

First, an up-projection matrix multiplies the input. Each row can be viewed as a direction used to probe the vector, and a learned bias is added to the result.

Suppose one row is the sum of the Michael and Jordan directions. Under the toy assumptions, its dot product reaches two for the full name and at most one when either part is missing. A bias of minus one moves the threshold.

A linear operation alone cannot isolate the combination cleanly. ReLU supplies nonlinearity by replacing negative values with zero and leaving positive values unchanged. In this constructed example the resulting activation acts like an AND gate for the two name features.

These intermediate values are often called neurons. A neuron is active when its output is positive. Real models may use a smoother activation such as GELU instead of the ReLU used for this illustration.

The down-projection maps the intermediate vector back to embedding size. Think of each matrix column as an output direction scaled by its corresponding neuron’s activation; the scaled columns are summed, along with a bias.

If the relevant column points in the basketball direction, activating the toy neuron adds that feature to the result. The residual connection adds this result to the input vector, associating the name with basketball.

The full block is therefore two learned matrix transformations, with biases and a nonlinearity between them, applied separately at each position. This is a familiar neural network embedded within the larger transformer.

For the GPT-3 dimensions used in this series, the up projection has about 604 million parameters and the down projection has roughly the same number. Across 96 MLP blocks, that is about 116 billion parameters, roughly two-thirds of the 175-billion-parameter model.

The matrix interpretation is exact, but the neat single-feature neuron in the example is not generally what researchers find. A neuron can participate in multiple features, making the actual computation harder to interpret.

Superposition is a proposed explanation: instead of requiring each feature direction to be exactly perpendicular to every other, a model may tolerate interference and use many nearly perpendicular directions.

In an n-dimensional space there can be at most n mutually perpendicular directions. In high dimensions, relaxing perpendicularity allows far more nearly perpendicular directions, although their overlap introduces noise.

I illustrate this by placing 10,000 vectors in 100 dimensions and optimizing their pairwise angles. High-dimensional geometry allows many more directions than dimensions; the exact capacity depends on the tolerated angle and interference.

A consequence of the Johnson–Lindenstrauss lemma explains the rapid growth in how many nearly perpendicular vectors can fit as dimensionality increases. This geometric capacity makes superposition plausible, but does not by itself prove how any particular model represents its knowledge.

The same idea can apply to the MLP’s intermediate activations: a feature might be represented by a combination of neurons rather than a single neuron. Sparse autoencoders are one tool researchers use to try to extract such features.

The next chapter will examine training, including the language-model loss, reinforcement learning from human feedback and scaling laws. The video links further reading on factual recall, superposition and interpretability.