Intelligence is compression

Karen Hao reports in Empire of AI (chapter 5) that while Alec Radford was teaching a transformer to predict the next word, the work that became GPT, Ilya Sutskever had a short way of explaining why it mattered: “Intelligence is compression.” A model trained to generate convincing text is forced to squeeze what it sees about the world into its essence. In a 2016 memo, Hao writes, Sutskever went further and argued that compression was the only thing needed for general intelligence. OpenAI’s public version borrowed Richard Feynman’s line, “What I cannot create, I do not understand”, which Hao calls “heavily anthropomorphized”.

I think the slogan is right, and older than OpenAI. The argument goes like this:

• To compress data well, you have to predict it. A compressor that knows what comes next only has to store the surprises.
• To predict text well, you have to model whatever produced the text: grammar first, then facts, then the reasoning and intentions of the people who wrote it.
• So the better the compression, the more of the world the model has had to capture. A model that compresses everything people have written down cannot help learning a great deal about people.

This is not a metaphor. Prediction and compression are the same problem in information theory, which is why Marcus Hutter has offered a prize since 2006 for compressing a fixed slice of Wikipedia, as a benchmark for intelligence. Next-word prediction is exactly that contest, run at the scale of the internet.

It is also half of a wager. The other half, in the same chapter, is Dario Amodei’s bet on the “pure language” hypothesis: that text alone is enough to learn from. Put together, they say that compressing everything humans have written is a path to general intelligence. That is my reading, not Hao’s, and it is the bet the rest of the book is about. Whether it is enough is still open; I lean towards thinking a model also has to act, as I argued in in vivo, not in vitro. But compression explains why the bet got as far as it did, and it fits the larger conviction I share with Altman, that intelligence is an emergent property of matter arranged to predict.

More on the book in my review, An Empire of People.

References

• Karen Hao, Empire of AI (2025), chapter 5, “Scale of Ambition”.
• The Hutter Prize for lossless compression of human knowledge, prize.hutter1.net.