How Shannon Measured the Entropy of English

ZUSAMMENFASSUNG

In 30 Sekunden

3Blue1Brown explains how Claude Shannon moved from counting short letter sequences to human prediction experiments. The resulting estimate connects English’s compressibility with the ability to predict what comes next.

1Prediction and compressionShannon linked the compressibility of English to the probabilities of its next characters.
2A human language modelBetty Shannon guessed letters from a passage, allowing correct guesses to be replaced by dashes.
3Counting guessesShannon expanded the experiment by recording how many guesses people needed and used the results to estimate probabilities.
4An estimate of one bitWith at least 100 characters of context, he estimated English might in principle be compressed to around one bit per character.

Das Wichtigste

A predictable next character carries less new information and can be compressed more efficiently.
Counting short sequences cannot reliably measure the predictive value of long contexts that rarely recur in a sample.
Shannon used human guesses to estimate probabilities and suggested English could approach one bit per character given substantial context.
Modern compression can use engineered predictive models rather than only measuring human prediction.

ThemenMathematics AI History

VOLLTEXT

Den bearbeiteten Text nach Kapiteln lesen

Transkript zeigen

Nur das Transkript

From letter counts to human prediction

Short-sequence statistics become sparse as context grows. Shannon turned to a human predictor to explore the information in longer contexts.

Claude Shannon connected the compressibility of English to how predictable its text is. His early experiments counted which characters followed short sequences: for instance, scanning books for every occurrence of “TH” and tabulating what came next.

That approach breaks down for long sequences, especially ones that never appear in the sample. Yet longer sequences provide more context, making the next character more predictable and the text more compressible. Shannon therefore needed another way to investigate those probabilities.

He turned to a language model available to him in the 1940s: his wife, Betty. As the story goes, he asked her to predict each new letter in a passage. When she guessed wrong he wrote down the correct letter; when she guessed right, he put a dash. The resulting string had fewer letters but, given a duplicate of the same predictor, could still be used to reconstruct the text.

This was a start, but Shannon’s formal definition of information required the probabilities assigned to the next letter. In his paper “Prediction and Entropy of Printed English,” he described experiments with more people that recorded how many guesses each needed to find the correct letter. He combined the number of guesses with short-sequence statistics to estimate the probabilities implicit in their predictions.

Estimating entropy from guesses

Shannon recorded the number of guesses needed and combined that information with short-sequence statistics. The estimate suggests a connection between compression and intelligent prediction.

FOLIE 1 / 1Shannon’s entropy estimateThe displayed paper and data table accompany the estimate derived from human guesses.

Given at least 100 characters of context, Shannon estimated that English might in principle be compressed to around one bit per character. To make that estimate he had to go beyond simple data analysis and probe something intelligent. Today, the speaker argues, compression near this limit involves trying to engineer predictive intelligence; this clip comes from a longer series exploring that connection.