Its magic crumbles away

Karen Hao opens and closes Empire of AI with the same sentence, from Joseph Weizenbaum’s 1966 paper on ELIZA: once a program’s inner workings are explained in language plain enough to induce understanding, “its magic crumbles away.” The observer, Weizenbaum went on, says to himself that he could have written it.

ELIZA was a simple procedural program that matched keywords and turned the user’s sentences back into questions, and Weizenbaum built it partly to show how easily people fall for an illusion of intelligence. They fell for it anyway. Psychiatrists talked about automated therapy, and Weizenbaum spent much of the rest of his career trying to deflate what he had started (chapter 4). For ELIZA the sentence is exactly true: explain the trick and nothing is left.

Hao uses it as her remedy against the influence of the AI companies: teach people how the technology works, and the hype loses its hold. I agree with the remedy. Explaining how these systems work is the most useful thing anyone writing about them can do, and it is most of what this site tries to do.

But with large language models the magic does not crumble when you explain it, and that is the interesting part. The procedure is simple enough to write down: predict the next word, adjust the weights, repeat over most of what people have written. Knowing that does not let you say you could have written the result, because the program you could write is not the one that runs. What the model learned lives in billions of weights that nobody wrote and nobody can yet read in full. The explanation of the method is plain; the explanation of the behaviour is still a research field.

So I would put Weizenbaum’s sentence to a test rather than use it as a remedy. If explaining a system makes its abilities disappear, it was a trick. If the explanation leaves the abilities standing, something real is there, and the honest response is to keep explaining, including where we cannot yet. That is the line between hype and capability, and it is why I take seriously the idea that intelligence is compression, and that it may be an emergent property of matter.

More on the book in my review, An Empire of People.

References

• Karen Hao, Empire of AI (2025), epigraph, chapter 4, “Dreams of Modernity”, and the epilogue.
• Joseph Weizenbaum, “ELIZA: A Computer Program for the Study of Natural Language Communication Between Man and Machine”, Communications of the ACM 9(1), January 1966, pp. 36 to 45, doi:10.1145/365153.365168.