Reinventing Entropy: Why Prediction Helps Compress Language

How much can text be compressed? The 3Blue1Brown narrator starts with ASCII’s eight bits per character and asks what determines the ultimate limit when an encoding exploits patterns in language.

Claude Shannon’s information theory offers a way to answer the question. Its mathematics also appears in language-model training, where next-token prediction is usually optimized with cross-entropy loss.

The narrator introduces the relationship between prediction and compression: a good probability model can support efficient coding. He treats “compression is intelligence” as a provocative claim rather than a precise theorem, since intelligence itself is difficult to define.

For a warm-up, imagine a robot receiving independent movement instructions: up with probability 1/2, down 1/4, and left and right 1/8 each. Bits are costly to send, so the goal is to encode these instructions efficiently.

A straightforward code gives each of the four directions two bits. The robot can decode by dividing the stream into pairs, but the code does not exploit the fact that some directions are more common.

A second code uses 0 for up, 10 for down, 110 for left and 111 for right. Its expected length is 1/2 × 1 + 1/4 × 2 + 1/8 × 3 + 1/8 × 3 = 1.75 bits per instruction.

Variable-length words need unambiguous boundaries. Here the robot reads until it has a complete codeword: 0 immediately means up; 10 means down; 110 and 111 identify left and right.

The key property is that no complete codeword is the prefix of another. This prefix-free code can be decoded as the bits arrive without guessing where one instruction ends.

A binary tree visualizes the constraint. Assigning 0 consumes all codewords beginning with 0; assigning 10 consumes a quarter of the space, while 110 and 111 each consume an eighth. Their shares match the instructions’ probabilities.

A third student asks what a perfect compressor would produce. The narrator builds intuition from fair, independent random bits: if the output retained predictable patterns, there could be an opportunity to compress it further.

In this carefully chosen robot example, each output bit behaves like an independent fair coin: half of messages start with 0, and conditional choices among the remaining words preserve the same balance.

Why can fair random bits not be compressed on average? For n bits there are 2^n equally likely strings. In a lossless, uniquely decodable scheme, shortening some possibilities requires making room by lengthening others.

The binary-tree picture makes the trade-off visible: assigning a shorter word takes more coding space and pushes other words deeper. For equally likely messages, equal-length codes are optimal in this setting.

If an event of probability p receives an ideal code length n, the relationship p = 2^(-n) gives n = −log₂(p). Shannon calls this quantity the information, or surprisal, of the event.

The formula remains useful even when p is not a power of two and the ideal value is fractional. Rare events carry more information in this sense; events close to certain carry little.

A fractional information value does not mean a physical bit can be split in isolation. The coding interpretation concerns average lengths across possible messages, or long sequences encoded together; an individual message can sometimes be assigned a shorter word at others’ expense.

Language adds two complications: probabilities depend on preceding context, and they rarely line up neatly with powers of two. A model may give one probability for the next letter after a particular phrase and a different one in another context.

The probability of a whole phrase is the product of its successive conditional probabilities. Taking the negative logarithm turns that product into a sum, so the information of the full message is the sum of the information assigned to each next letter.

Those fractional contributions can accumulate into the length of an efficient code for an entire message, rounded to whole bits. The narrator promises a later installment that shows a concrete compression algorithm.

The hard question is where the probabilities come from. A language model’s estimates need not be the true distribution of language, and even defining a single “true” distribution for language is not straightforward.

Shannon first studied n-grams: count what letters follow a short sequence such as “th” in books. But long contexts quickly become sparse, including combinations that never appear in the sample.

He also used a human language model. In one story, Shannon asked his wife Betty to guess the next letter of a passage and recorded correct guesses differently from errors. If an identical guesser repeated the process, that reduced record could reconstruct the text.

The experiment illustrates how predictability can support compression, though people do not necessarily make identical guesses twice. Shannon later used the number of guesses needed for each letter to estimate probabilities more systematically.

The key point is that Shannon was probing a model of language inside a person’s brain, not merely counting characters in a corpus. Modern language models provide another kind of predictive system, though their probabilities are still model estimates.

Entropy asks for average information per symbol. For independent symbols drawn from a fixed distribution, it is H = −Σ pᵢ log₂ pᵢ. In the robot example the expected code length is exactly the entropy, 1.75 bits per instruction.

The visualization gives each probability a bar of width p and height −log₂ p. The total area is the weighted average information. A more even distribution has higher entropy than one dominated by a likely outcome, for a fixed set of possibilities.

In the lossless coding setting, entropy gives a lower bound on expected bits per symbol, and Shannon’s noiseless coding theorem says block coding can approach that limit arbitrarily closely under its assumptions.

Natural language needs a more general quantity: entropy rate, which accounts for context across sequences and averages information over possible messages. It is much harder to calculate because the underlying distribution of language is not known exactly.

The narrator reports that Shannon estimated roughly one bit per English character when human guessers had at least 100 preceding characters of context. It is an estimate tied to that experiment, not a universal constant for all texts.

The next installments will show how a language model can support compression and explain cross-entropy in model training. This first part establishes how probabilities, information and entropy arise from the coding problem.