Joseph Weizenbaum published ELIZA in 1966. A program of a few rules that imitated a psychotherapist by echoing the user's own sentences back. His own secretary asked him to leave the office so she could talk to the machine alone. Sixty years on, the large language models do the same thing with a hundred million users a week. The illusion of thought is not a technical achievement. It's a human reflex. And the industry has spent six decades perfecting the reflex, not the thinking.
The Program That Fit in Two Hundred Lines
In 1966, Joseph Weizenbaum wrote about two hundred lines of code in MAD-Slip, a language of the time. He called it ELIZA, after the flower girl in Pygmalion. The main script, DOCTOR, imitated a Rogerian psychotherapist (a therapeutic school that limits itself to giving the patient back their own words reworded). It read you a sentence. Identified keywords. Reworded them as a question. And lobbed the ball back to you.
If you typed "my mother hates me," the program replied "tell me more about your family." If you typed "I'm sad," it replied "why are you sad?" It understood nothing. It did pattern matching against a table of rules and applied a syntactic substitution. It was, literally, a parlor trick implemented in sixty pages of printout.
And it worked.
It worked so well that Weizenbaum's own secretary, who had watched him write the code line by line, asked him to leave the office so she could talk to the machine in private. He recounts it himself in Computer Power and Human Reason, ten years later, still baffled. And he leaves the diagnosis in writing: "extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people." He doesn't say naive. He doesn't say gullible. He says quite normal.
That sentence is almost sixty years old. It's still the most relevant datum ever produced about human-machine interaction, and the whole industry has spent sixty years pretending it never read it.
The Trick That Wasn't a Trick for the Brain
ELIZA had no model of the world. It had no memory between sentences beyond a few variables. It had no semantic representation. What it had was a repertoire of rewording templates that gave the sense that someone was listening. And the human brain, faced with text that seemed addressed to it, did what it has done since it existed. Attribute agency.
Daniel Dennett called this the intentional stance in the 1987 book of the same name (a mental strategy by which we predict a system by assuming it has beliefs and desires, instead of reconstructing its mechanics). When something behaves as if it had a mind, treating it as if it did is cheaper than working out whether it does. The brain adopts that stance by default, because it costs nothing and almost always gets it right when the other party is made of flesh. Detecting minds was a matter of life and death for our ancestors. Detecting the truth about the nature of those minds was never a problem under evolutionary pressure.
The reflex is old, automatic, and operates below the level where critical examination can get in.
What Weizenbaum discovered in 1966 is that the reflex fires with code too. No flesh required. Well-formed text that seems to address the reader is enough. If the system says "I understand," the brain registers understanding. If the system says "tell me more," the brain registers interest. The word is signal enough. The brain doesn't audit the provenance.
What Turing Promised Without Knowing It
Sixteen years before ELIZA, Alan Turing had published in Mind an article that has been misread for seventy-six years. The 1950 paper proposes the imitation game: if a machine can hold a textual conversation without a human judge telling it apart from a person, we should treat the question "can machines think?" as badly framed and replace it with the operational question. Turing's technical sentence is cautious. The cultural reading was triumphalist. If a machine passes the test, it thinks.
Weizenbaum showed in 1966 that the inference was invalid.
ELIZA didn't pass the Turing test in a sustained conversation, but it passed something close. It produced in the human interlocutor the subjective conviction of conversing with an agent. And the judge's subjective conviction was never good proof of anything, because the judge is the system's most easily fooled organ. What the test measures, looked at coldly, is not the machine's intelligence but the judge's propensity to attribute it. Which is another thing. It's the magnitude of the anthropomorphizing reflex, not the magnitude of the thought on the other side.
This isn't an academic detail. It's the whole crack through which, sixty years later, the present economy of language models slips.
From Two Hundred Lines to Trillions of Parameters
A modern LLM (large language model) doesn't resemble ELIZA on the inside. Hundreds of billions of parameters. Corpora of trillions of lexical units. Attention mechanisms that capture long dependencies. Post-training with human feedback that tunes the tone. It's a technical beast ELIZA is not. To deny it would be absurd.
And yet.
The cognitive structure of the human user hasn't changed at all between 1966 and 2026. The piece that produces the illusion of thought isn't in the machine. It's in the reader. The reflex of attributing agency, the one that fills gaps by assuming the other understands, the one that takes fluency as proof of comprehension —those reflexes haven't been updated. They operate with the LLM exactly as they operated with ELIZA. What has changed is the quality of the stimulus, not that of the receiver.
The Stochastic Parrot Sings Opera
Bender and Koller, in their ACL 2020 paper titled Climbing towards NLU, said it in technical terms. A system trained only on linguistic form has no access to meaning, however well it predicts the next word. A year later, Bender, Gebru, McMillan-Major and Shmitchell published On the Dangers of Stochastic Parrots. The word "parrot" angered half the industry, and it angered them precisely because it pointed at the uncomfortable spot. What the model produces is a statistically plausible recombination of observed patterns. What the user interprets is comprehension. The distance between the two things is bridged by the same old reflex.
That the parrot is much bigger doesn't change that it's a parrot. It changes that it now sings opera.
The Deliberate Design of the Reflex
This is where it gets ugly, because pretending neutrality is already a form of complicity.
Today's systems aren't statistical machines that produce, as an involuntary byproduct, the illusion of intelligence. They're machines designed explicitly, in their post-training layer, to maximize that illusion.
Reinforcement learning from human feedback (in English, RLHF, a technique where human annotators rate the model's answers and the system learns to produce the ones they prefer) trains the model to produce answers the raters prefer. And what do they prefer? Answers that sound kind, confident, flattering to the interlocutor, delivered with the cadence of someone who knows what they're saying. The model learns to optimize the appearance of knowing. It doesn't optimize knowing. It optimizes the external signal of knowing, because the external signal is what the rater scores. It's Goodhart's law applied to thought (when a measure becomes a target, it stops being a good measure).
Calibrated Voices, Affects Bolted On
On top of this comes personality design. Kind voices. First names. Colloquial tones. Simulated memory between sessions. The ability to say "I'm glad to talk to you again" when technically there's no affection to speak of.
There are whole teams calibrating the degree of simulated affection the user tolerates without discomfort, and the degree below which the product loses stickiness. That curve exists. It's measured. It's optimized.
It isn't an oversight. It isn't an inevitable consequence of machine learning. It's a product decision. You could build a useful assistant that presented itself as what it is: a statistical text completer, with no first name, no affective tone, no "I think" or "it seems to me," no false apologies. You could. It isn't done, because it produces worse retention. Simulated emotional fluency is the commercial lever. Calling it by its name is necessary.
The Trap of the Responsible User
The industry's standard defense, when it's pointed out that its products induce parasocial bonds with code, is the same defense the alcohol or sugar industry uses. The responsibility lies with the adult consumer. Let them educate themselves. Let them learn to use the tool. Let them tell apart what the system actually does from what it seems to do. Let them not let the simulacrum go to their head.
The argument has a problem. It's the same problem Weizenbaum pointed out in 1966.
What You Can't Ask of an Eye
The anthropomorphizing reflex isn't a cognitive bias correctable with a three-hour course. It's a structural property of the human brain, installed by evolutionary pressure, that operates below the level of consciousness. Asking the user to "stay critical" in front of a system designed to activate that reflex is like asking someone not to go blind staring at the sun. The blindness isn't a decision.
If the illusion of thought is produced by a two-hundred-line program in 1966, with people who have seen the code, demanding more critical capacity of the 2026 user in front of systems with trillions of parameters and experience-design (UX) teams optimizing emotional fluency is shifting onto the brain a cost it biologically cannot bear. The human brain wasn't wired for that. The load can't be shifted onto it because the load isn't shiftable. It's like asking the eye not to process light.
What can be shifted is the responsibility of design. The system that exploits the reflex is modifiable. The brain that suffers it is not. Any regulatory or ethical framework that ignores this asymmetry isn't naive. It's self-interested.
What Weizenbaum Said and No One Wanted to Hear
In 1976, ten years after ELIZA, Weizenbaum published Computer Power and Human Reason. A short book, written by the program's own inventor, devoted to explaining why his program should never have been mistaken for a therapist.
The book was poorly received at MIT, where they accused him of betraying the guild. It was ignored by the industry that, over the next fifty years, built the present business of conversational models. Weizenbaum died in 2008. He didn't live to see GPT-3. He lived long enough to see the pattern.
What he wrote, translated without ornament, is that it isn't ethically neutral to build systems that produce in the human being the illusion of being understood when they are not. The objection isn't technical. It's operational. The harm isn't in the machine. It's in what the machine activates in the person looking at it, and in the possibility that this activation is designed to maximize engagement, dependence or purchase.
Sixty years of industry have answered Weizenbaum the way you answer an annoying old man. Not by refuting him. By letting him talk to himself. Today people argue over whether the models "understand," whether they have "sparks of general intelligence," whether they pass this or that standardized test (benchmark). Everything is argued except the one thing Weizenbaum wanted to argue. That the illusion of comprehension is supplied by the observer. That the system is designed to amplify it. And that this asymmetry isn't an accident.
Who Looks at Themselves in the Mirror
Every time someone writes to a chatbot that they feel lonely, and the chatbot answers with a sentence calibrated by a retention-optimization system, and the person registers relief, the same thing is happening as in Weizenbaum's office in 1966.
The mechanism hasn't changed. The industry has had sixty years to perfect the reflex. Not to build thought on the other side, among other reasons because thought on the other side was never the goal. The goal was always for the reflex to fire harder.
A two-hundred-line program was enough to make Weizenbaum's secretary ask him to leave the office. Imagine how little it takes today. No need to imagine it. It's happening, right now, with the hundred million people who every week sit down in front of one of these systems, people who receive from a system trained to please them something their human surroundings stopped giving them long ago, and who go back to the screen tomorrow because the screen's voice is exactly the voice their brain is wired to mistake for someone who listens.
The Mirror Polished for Six Decades
The machine doesn't think. The machine never thought.
ELIZA didn't think in 1966 and the large language models don't think in 2026. The line between the two runs through orders of magnitude, not through an ontological boundary. What has changed is the sharpness with which the mirror gives back the image of the one looking. It has the voice the user wanted to hear. It has the topics that interest them. It has the cadence that calms them. But it's still the mirror. What the user sees inside isn't anyone. It's themselves, given back by a surface polished for sixty years so the glass doesn't show.
Weizenbaum saw it in sixty-six and said it in seventy-six. The industry has, in the meantime, billed obscene amounts selling the reflex as if it were thought. And on top of it, the user is supposed to be at fault for not having told them apart.
Definitions
ELIZA. A program written by Joseph Weizenbaum in 1966 at MIT. About two hundred lines of code in MAD-Slip. Its best-known script, DOCTOR, imitated a Rogerian psychotherapist by rewording the user's sentences as questions, with no real comprehension of the content.
Turing test or imitation game. Alan Turing's 1950 proposal. A machine passes the test if a human judge, in a blind textual conversation, can't tell it apart from a person. Turing didn't claim that passing it was equivalent to thinking; that reading came later.
Intentional stance. A concept of Daniel Dennett's. A mental strategy by which we predict a system's behavior by assuming it has beliefs and desires, rather than reconstructing its inner workings.
Large language model, LLM. A statistical system trained on massive quantities of text to predict the most probable next lexical unit in a sequence. With no access to the world, no verification, no guarantee of coherence beyond statistical plausibility.
Reinforcement learning from human feedback, RLHF. A post-training technique where human annotators rate the model's answers. The system learns to produce the best-rated answers, which tends to optimize the appearance of competence more than competence itself.
Goodhart's law. The principle that, when a measure becomes a target, it stops being a good measure. Applied to RLHF, optimizing the external signal of knowing ends up replacing knowing.
Stochastic parrot. A phrase introduced by Bender, Gebru, McMillan-Major and Shmitchell in 2021 to describe language models. They statistically recombine observed patterns with no access to the meaning of what they produce.
References
Weizenbaum, J. (1966). ELIZA — A Computer Program for the Study of Natural Language Communication Between Man and Machine, Communications of the ACM 9, 36–45. The program's founding article. Available at https://dl.acm.org/doi/10.1145/365153.365168.
Weizenbaum, J. (1976). Computer Power and Human Reason. From Judgment to Calculation, W. H. Freeman. The creator's own critique of the confusion between conversational simulation and comprehension, cited in the sections on the ELIZA effect and the industry's defense.
Turing, A. M. (1950). Computing Machinery and Intelligence, Mind 59, 433–460. The text the imitation game comes from, cited in the section on the cultural misunderstanding of the test.
Bender, E. M. and Koller, A. (2020). Climbing towards NLU. On Meaning, Form, and Understanding in the Age of Data, ACL 2020. The technical argument on the impossibility of accessing meaning by training only on linguistic form.
Bender, E. M., Gebru, T., McMillan-Major, A. and Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. Can Language Models Be Too Big?, FAccT 2021. The origin of the phrase "stochastic parrot." Available at https://dl.acm.org/doi/10.1145/3442188.3445922.
Dennett, D. (1987). The Intentional Stance, MIT Press. The source of the intentional-stance concept used to explain the automatic attribution of mind.
OpenAI / TechCrunch (6 November 2023). OpenAI's ChatGPT now has 100 million weekly active users. Sam Altman announced the figure of a hundred million weekly active users at OpenAI's first developer conference; the source of the usage magnitude cited in the intro and the closing. Available at https://techcrunch.com/2023/11/06/openais-chatgpt-now-has-100-million-weekly-active-users/.
Para profundizar
Turkle, S. (2011). Alone Together. Why We Expect More from Technology and Less from Each Other. Basic Books. A study of anthropomorphization and affective bonding with machines; a natural read on the extension of the ELIZA effect into the present ecosystem.
Hofstadter, D. (1979). Gödel, Escher, Bach. An Eternal Golden Braid. Basic Books. A classic exploration of the illusion of emergent comprehension in formal systems.
También te interesa
- Recognizing Is Not Understanding, and the Border Between the Two Is Invisible
- The Problem of Defining Intelligence
- Talking to Something That Doesn't Exist
No comments yet
No comments yet. Be the first.
Leave a comment