Thought and language. Speaking well isn't the same as thinking well

For two hundred thousand years we've assumed that whoever articulates with elegance thinks with order. The assumption was operationally correct because the only system in our environment capable of producing sentences was another human. A system has just appeared that produces the sentences without the thinking part. And our wiring hasn't caught on.

A paper the industry would have preferred not to read

In July 2020, a researcher and a researcher, Emily Bender and Alexander Koller, presented at ACL a paper titled Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. The thesis said something the industry had spent months not wanting to hear: a system trained solely on the form of language has, a priori, no path to learning meaning. It learns the distribution of signs among themselves. It doesn't learn the connection between the sign and the thing outside the sign.

It's worth fixing the chronology, because it usually gets told wrong. GPT-3 had appeared two months earlier — the preprint Language Models are Few-Shot Learners is dated 28 May 2020, and API access arrived in June. Bender and Koller's paper got buried under the avalanche that followed. ChatGPT, the system that would carry the debate to the general public, didn't launch until 30 November 2022. Today, in 2026, with models several orders of magnitude larger and a thousand times more fluent, the argument still hasn't fallen. It's still the argument people most prefer not to name.

The syllogism that broke the wiring

There's an ancient human intuition, so deep we don't think of it as intuition but as fact: whoever speaks well thinks well. And the other way around.

The reflex is so automatic it has little to do with evidence. Everyone has met the eloquent fool and the brilliant stammerer. Everyone has watched the empty politician speak with perfect cadence for ten minutes and the competent engineer fail to finish a single sentence the whole meeting. We know the reflex fails. We apply it anyway, because the cost of judging what someone says without being swayed by how they say it is high, and almost nobody pays it.

A language model is the system designed to exploit that reflex at industrial scale. Optimized to predict the most probable next word against obscene amounts of human text, it ends up producing prose that triggers the automatic recognition: "this was written by someone who knows." No one wrote it. It was written by a function that assigns probabilities to sequences of tokens. The function is very good. The recognition, on our side, isn't a conscious choice; it's wiring.

The word without the thing

Bender and Koller put it as a thought experiment. Imagine a system trained solely on transcripts of conversations, with no access to anything else. No images, no sensors, no body, no world. Only the text. The system learns which sentences follow which sentences. What it doesn't learn, because it isn't in the data, is reference. The word "apple" appears next to "red," next to "tree," next to "bite." The system infers an internal geometry among those words. But the apple itself, the physical object you can pick up and bite, is outside its world. The word is a label with nothing labeled.

The typical objection, in 2026, is that current models are no longer only text. They're multimodal models. They've seen millions of labeled images, processed video, some pilot robots. Doesn't that solve the problem?

No. Bender and Koller's argument didn't say the problem was the absence of pixels. It said meaning isn't in the form, whether it's encoded in letters, in pixels, in audio samples or in proprioceptive signals. Adding modalities widens the model's formal space. It doesn't give it communicative intent. Intent is what an agent has when it needs something from the world and directs its actions toward getting it. It's what the hungry baby has when it cries. It's what the model, by construction, doesn't have: it needs nothing, nothing breaks for it if the sentence comes out wrong. Predicting the next token doesn't resemble being hungry.

The Chinese room still hasn't fallen

John Searle had told the same story forty-six years earlier with less paraphernalia. In 1980, in Minds, Brains, and Programs, he proposed the thought experiment of the Chinese room. A monolingual English speaker is locked in a room. He receives papers written in Chinese and has a rulebook, in English, telling him which sequence to output for each input sequence. He doesn't understand Chinese, he does zero comprehension. But his answers are indistinguishable from those of an educated Chinese speaker.

Does the system understand Chinese? The intuitive consensus is no. What's inside is symbolic manipulation. And a computer program, Searle said, is nothing but symbolic manipulation.

The argument has been refuted a thousand times and still hasn't fallen. The only way to answer it is to define understanding without falling into circularities, and nobody has done it. The strong-AI defender replies that you can't prove humans understand either. He's right on the detail and wrong on the move: turning the problem into one of symmetric impossibility doesn't prove the LLM understands, it proves that understanding is a rebellious concept. Which was, exactly, what Searle said.

Surface plausibility as a product

Let's put this on the ground. A current model summarizes scientific papers competently, translates above the average professional translator, drafts emails, proposals, code. In all that it's genuinely useful.

And at the same time, in the same paragraph, the same model will tell you the Eiffel Tower is in Rome, will get a three-digit sum wrong, will cite an invented ruling that any junior lawyer would catch in five seconds.

It's not an isolated failure. When what you optimize is the surface plausibility of the next token, what you get is surface plausibility. Sometimes it coincides with the truth because the truth was in the data. Sometimes it doesn't coincide, and the model has no way of knowing the difference, because the difference isn't in the form. It's in the reference. And reference is what it lacks.

If you use an LLM to draft emails, not much happens. But if you're using it, and you are using it, to summarize medical documents before a clinical decision, to prepare reports a judge with little time will sign, to evaluate candidates for a job, you're delegating the reading to a system whose only proven competence is producing text that seems correct. Not text that is correct. The difference, in a prescription or a ruling, is the only one that matters.

What it gives us, what it takes away

The trap isn't only cognitive. It's economic. Fluency is now an asset: producing text that sounds like an expert is cheap, fast, and at first glance indistinguishable from text produced by an expert. Don't assume this isn't the case with what you're reading.

There's a 2025 study from the MIT Media Lab, Your Brain on ChatGPT, that measures something fairly ugly and fairly predictable. When a group of people write essays with the assistance of an LLM over weeks, their later performance without the assistance drops measurably. The authors call it "accumulated cognitive debt": the system advances you capacity, and you pay for it later in your own diminished skill.

We're shifting scenes imperceptibly and with no way back. AI is going to be, already is, an indispensable tool. We'll kill for it: governments will compete for control of its infrastructure, companies will fight over its models the way they used to fight over oil. It's worth asking bluntly what it gives us and what it takes away.

The obvious comparison, which doesn't hold

Did the calculator make us stupid? Not entirely. But we lost the capacity for mental arithmetic. Most adults today can't subtract two-digit numbers in their heads without hesitating; thirty years ago any shop assistant could. Does that loss matter? In many contexts, no; in some, yes.

Did computers make us stupid? Not that either. But an entire generation has grown up unable to find its way without GPS, unable to hold a phone number in memory. There was loss, offset by new capacities; calling that "losing nothing" is a trick.

With LLMs the substitution is of a different kind. We're not delegating arithmetic or phone-number memory. We're delegating the articulation of thought. And the articulation of thought, unlike calculation, was the activity that produced thought itself. Writing isn't writing down what's been thought; writing is thinking. Whoever delegates the writing delegates the only forge of thinking the species has invented. That's a net loss. Denying it is propaganda.

The editor who never becomes an editor

There's a kindly version of this argument, the ad version: the tool frees the human from mechanical tasks and lets them concentrate on the qualitative. The fairy-tale part is assuming the skill of critically evaluating the model's output stays intact while the skill of producing the output is outsourced. It doesn't. Criticism is trained by practicing production.

A competent editor who has spent thirty years writing can review a machine-generated text and detect what fails. An intern who has read more machine-generated text than original human text will never become that editor, because the road to it ran through twenty years of getting it wrong while writing. We're cutting the staircase in half and saying that since there are still people up top, all is well. When those people retire, there'll be nobody to climb.

The two-hundred-thousand-year syllogism

Why is it so hard for us to separate fluency from thought? Because evolutionarily we never needed to.

For two hundred thousand years, the only system in our environment capable of producing articulate sentences was another human. If something spoke coherently, it was human. If it was human, it had a mind. The syllogism, speaking implies thinking, was operationally correct one hundred percent of the time. It no longer is. The syllogism has broken and our hardware hasn't registered it.

When a model answers you with a well-built sentence, your brain runs the old syllogism. You process form and issue a verdict on substance. It's the same shortcut we've been using since the species started speaking, only this time it leads to a cliff.

Who pays for the syllogism to keep working

Here it turns politically ugly. There are actors with an active interest in the shortcut continuing to work. A company selling general reasoning capacity doesn't need to prove it. It's enough that the appearance be convincing and that the audience keep running the old syllogism. The indefinition is a product. The fluency is marketing. That's why almost everyone who spells it out is off the payroll of the big labs.

Bender, Gebru, McMillan-Major and Shmitchell published in 2021, at FAccT, On the Dangers of Stochastic Parrots, where they called LLMs stochastic parrots: systems that reproduce form without access to understanding, at a scale that makes them pass for interlocutors. Timnit Gebru left Google in December 2020 in a dispute over that very work; she maintains she was fired, Google described it as accepting her resignation. The signal was clear: writing what a system actually does, instead of what the company needs it to seem to do, has material consequences.

The part you bring

It's easy to be outraged at the companies. But the anthropomorphizing reflex wasn't invented in Mountain View. Each of us brings it.

When a model answers you with elegance, a part of you wants to believe it has understood your question, because the alternative —that a system without understanding has produced you such a satisfying answer— is slightly terrifying. It implies that your whole life, when you thought you were recognizing thought in whoever spoke well, you might have been recognizing only form. It implies that the criterion by which you evaluate colleagues, teachers, bosses and experts was always fallible.

The discomfort isn't about the model. It's about looking at the model and realizing how the criterion works by which you've been looking at humans your whole life.

The decisions made with the old shortcut

The question that remains isn't whether the model thinks. It's harsher. What decisions, in your life, in your work, in the institutions that affect you, are already being made under the implicit assumption that articulating well implies understanding well? The doctor who delegates the first draft of the diagnosis to an assistant. The judge who asks the system for a summary of the case. The teacher who corrects essays with the model's help and stops reading them himself. The journalist who signs with their name what a system generated.

Speaking well doesn't imply thinking well. It never fully implied it; we accepted it as a tolerable approximation because there was no alternative. Now there's a machine that produces the speech without the thinking part, and it forces us to separate two things we'd kept joined by habit for two hundred thousand years. The question is whether you, listening to the machine, still know how to tell the difference. And whether, when you next hear an eloquent human, you'll still know not to automatically grant them what you'd been granting without noticing. That concession, free and ancient, was the shortcut. Someone has just come to collect.

Definiciones

Stochastic parrots. Expression coined by Bender, Gebru, McMillan-Major and Shmitchell in 2021 to describe large language models: systems that reproduce patterns of linguistic form without access to understanding what they say, assembling text according to statistical probability learned in training.

Chinese room. Thought experiment proposed by John Searle in 1980. A man who doesn't understand Chinese, locked in a room with a rulebook in his own language, can produce Chinese answers indistinguishable from those of a native speaker by manipulating symbols without understanding them. The argument holds that running a program, however sophisticated, is syntactic manipulation and doesn't generate semantic understanding.

Form vs communicative intent. A central distinction in Bender and Koller's argument. Form is the observable sequence of signs (letters, words, sentences). Communicative intent is the internal state of the agent that needs something from the world and produces signs to get it. A system trained only on form has no way to recover the intent that originated that form, because the intent isn't in the data.

Multimodal model. AI system trained simultaneously on several types of data (text, image, audio, video, sensory signals) rather than just one. Multimodality widens the model's formal repertoire but, by Bender and Koller's argument, doesn't by itself solve the problem of reference: it adds more forms, not meaning.

Referencias

Bender, E. M. & Koller, A. (2020). Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. Proceedings of ACL 2020. https://aclanthology.org/2020.acl-main.463/. Central argument of the article: a system trained only on form has, a priori, no way of learning meaning.

Searle, J. R. (1980). Minds, Brains, and Programs. Behavioral and Brain Sciences 3:417–457. Source of the Chinese room thought experiment, brought back here forty-six years later.

Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Proceedings of FAccT 2021. DOI: 10.1145/3442188.3445922. Origin of the expression "stochastic parrots." Timnit Gebru's departure from Google in December 2020, tied to this work and disputed between the parties (a firing per Gebru, an accepted resignation per Google), is reported in MIT Technology Review (4 December 2020): https://www.technologyreview.com/2020/12/04/1013294/google-ai-ethics-research-paper-forced-out-timnit-gebru/.

Kosmyna, N., Hauptmann, E., Yuan, Y. T., Situ, J., Liao, X.-H., Beresnitzky, A. V., Braunstein, I. & Maes, P. (2025). Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task. MIT Media Lab. Preprint at arXiv:2506.08872. Project site: https://www.media.mit.edu/publications/your-brain-on-chatgpt/. Source of the data on accumulated cognitive debt in habitual users of AI assistants.

Para profundizar

Marcus, G. and Davis, E. (2019). Rebooting AI. Building Artificial Intelligence We Can Trust. Pantheon. A direct critique of the overlap between fluency and understanding in neural systems.

Hofstadter, D. (1979). Gödel, Escher, Bach. An Eternal Golden Braid. Basic Books. Formal systems that produce structure without accessing meaning.

Searle, J. R. (1992). The Rediscovery of the Mind. MIT Press. A later reformulation of the argument against strong AI, after the Chinese room.

También te interesa

En otros sitios

#intelligence #reasoning #anthropomorphism #ai-writing #papers