In 30 Sekunden
Grant Sanderson explains why an event with probability p carries −log₂(p) bits of self-information, using successive reductions of equally likely possibilities.
- 1Halving possibilitiesHalving equally likely possibilities yields one bit.
- 2ProbabilityAn event of probability p has self-information −log₂(p) bits.
- 3The logarithmEntropy averages self-information across possible events.
Das Wichtigste
- Halving equally likely possibilities yields one bit.
- An event of probability p has self-information −log₂(p) bits.
- Entropy averages self-information across possible events.
Den bearbeiteten Text nach Kapiteln lesen
Information in bits
From halving possibilities to a logarithmic formula.
I introduce the bit as a unit for the information associated with an observed event. This short comes from a longer lesson that applies information theory to Wordle.
If an observation reduces equally likely possibilities by a factor of two, it provides one bit of information. Reducing them by a factor of four gives two bits, and by a factor of eight gives three.
For an event with probability p, the number of bits I satisfies p = 2^(−I). Equivalently, 2^I = 1/p.
Taking the base-two logarithm gives I = log₂(1/p) = −log₂(p). Less probable events carry more self-information under this definition.
This is the self-information of one observed event. Entropy is the expected self-information over a probability distribution, a related but distinct quantity.
Mehr aus diesem Kanal
40:44EN
3Blue1Brown· 16. Okt. 2026Grant Sanderson: There's never been a better time to go into mathGrant Sanderson, Constantin Kogler · 3Blue1BrownStudents fear AI has solved mathematics. Grant Sanderson argues the opposite: even if machines prove theorems better than we do, the institution of math stays valuable—and this is a rare moment when newcomers can reshape it.
51:55EN3Blue1Brown· 18. Sept. 2026The last IMO problem AI could not solveGrant Sanderson · 3Blue1BrownGrant Sanderson walks through IMO 2025 Problem 6 — a tiling puzzle that frontier models missed — and the visual patience it takes to prove a construction is optimal.
0:55EN3Blue1Brown· 24. Juli 2026The 64 Sugar Cubes Puzzle: Six Equal DistancesA 4×4×4 block has three colors; prove that six same-colored pairs share a distance.
33:51EN3Blue1Brown· 16. Juli 2026What Cross-Entropy Connects: Compression, Language Models and KL DivergenceGrant Sanderson · 3Blue1BrownGrant Sanderson builds cross-entropy from code lengths and uses it to explain next-token training and distillation.
2:25EN3Blue1Brown· 12. Juni 2026How Shannon Measured the Entropy of EnglishA guessing game helped Claude Shannon estimate how much information English text contains, and why prediction is central to compression.
1:02EN3Blue1Brown· 10. Juni 2026A Robot Walk and the Puzzle of Optimal Binary EncodingHow few bits does it take to send directions that occur with unequal frequencies?