In psychology, the word ‘heuristic’ refers to a kind of mental shortcut, an equivalency that people make in order to assist with a rapid evaluation or decision. A typical heuristic might be a simplified attribute standing in for a more complex one, or a perceptible characteristic standing in for a hidden one. Idiomatically, you might call it a ‘rule of thumb’.
We need heuristics, because the world is incredibly complex, and if we didn’t simplify our mental models we’d never be able to make a decision about anything. But the risk with any heuristic is that salient, even essential points can get lost in the simplification and substitution, leading to a false outcome.
“Language skill indicates intelligence,” and its logical inverse, “lack of language skill indicates non-intelligence,” is a common heuristic with a long history. It is also a terrible one, inaccurate in a way that ruinously injures disabled people. Now, with recent advances in computing technology, we’re watching this heuristic fail in ways that will harm almost everyone.
The Double-Sided Heuristic Coin
Heads …
When I discovered — a few years ago now — that a ridiculous number of unsolved mysteries about my life experience could be explained by ‘autism’, I responded in the most autistic way ever: by hoovering up every crumb of information about autism that I could find. Because it was pretty quickly obvious that the academic study of autism by allistic
researchers had resulted in a giant pile of unscientific, erroneous, and damaging bullshit,
I leaned heavily on memoirs and other first-person accounts.
My reading eventually expanded to include stories from autistic teens and adults with profound speech and communication challenges, ones that I myself never experienced. Some autistic kids are simply very delayed: they don’t speak at all for many years, but then later develop spoken language indistinguishable from allistic children. Others have ‘unreliable speech’, where they have imperfect mental control over the words and sounds they utter. And still others have verbal apraxia, a brain-body disconnect that impedes their ability to shape their mouths in the intricate ways that speech requires.
The lives of these kids changed only when a parent or other adult in their life took exceptional steps to provide communication options other than speech.
Of course, the only first-person stories available to me as I researched were from autistic people who eventually did find a way to communicate. Some had learned to type on a regular keyboard, with varying levels of ease or difficulty depending upon their physical challenges. Others use a computer tablet with specialized software (called AAC, for ‘Augmentative/Alternative Communication’). Many autistics with severe motor dyspraxia have found success communicating with a letterboard, laboriously spelling out sentences one letter at a time, often with a support person to help them focus and persist despite a body that doesn’t obey their brain’s commands.
Over and over again, too many times to be anything other than a pervasive pattern, I saw the same story: in blog posts and videos and published memoirs, autistic teens and young adults described living for a decade or more without any way to communicate, while people around them assumed they were intellectually deficient. Kids who heard and understood every time someone referred to them as stupid or simple, who were in terrible pain but had no way to explain the problem, who could read at or above grade level but were forced to repeat the same basic alphabet drills for years. It was horrifying to imagine, and repeatedly broke my heart.
The lives of these kids changed only when a parent or other adult in their life took exceptional steps to provide communication options other than speech. But this is still a rare outcome, because the primary sources of information for parents mention none of these alternatives, and the primary therapy promoted today for autistic kids with motor and communication disabilities assumes they have less sentience than a dog.
We don’t know exactly how many other nonspeaking
autistics have never been offered specialized AAC software or letterboard assistance or whatever communication channel they happen to need, and so spend their entire lives trapped in frustrated, tedious, heartsick silence.
But the prevalence of these stories suggests that many — perhaps even most — nonspeaking children have fully ‘normal’ intelligence, as intelligence is popularly understood and typically measured.
Some of them are undeniably brilliant. There are definitely autistic children who also have intellectual disabilities, but not nearly as many or as severe as people assume … because the rest of humanity keeps using the language fluency heuristic.
You know who else has been suffering under this same flawed equivalency for all of known human history? Deaf people.
Aristotle, for one, baldly stated that since people born deaf lacked access to spoken language, they were “senseless and incapable of reason”. The award-winning play and movie The Miracle Worker cemented Helen Keller — arguably the most famous Deaf person in the world — in the minds of generations of viewers as a “wild animal” who “clawed and struggled against all who tried to help her” before Annie Sullivan came along and taught her what language was. (It’s a completely false narrative — Helen did not behave like a wild animal; she was already communicating with her own original word-signs before Sullivan came along.) But Deaf people have been treated like animals (or worse) in many times and places, because hearing and speaking people assumed they were unintelligent.
The equivalence gets baked right into the very definitions of our words, going at least as far back as classical Latin, where the word ‘surdus’ means ‘deaf, silent, stupid’, as if those concepts were inextricable and interchangeable. Separately, the English word ‘dumb’ (and its German predecessor) also came to mean both ‘speechless’ and ‘stupid’.
For the first time in history, we’re watching the rapid, massive, and egregious failure of this underlying heuristic affect all of human society.
Or just look at U.S. President Biden, who’s disclosed that his childhood stutter caused people to assume that he was “stupid” and “totally incompetent”. The examples are endless, but you see the pattern: anyone who struggles with the culturally typical form of language is seen as simple-minded at best and sub-human at worst.
This has been a recurring problem for at least thousands of years; it rarely improves or even gets much attention, because it has only ever affected a small, inherently disadvantaged subset of the population. What’s interesting about the current moment is that — for the first time in history — we’re watching the rapid, massive, and egregious failure of this underlying heuristic impact all of human society.
… and Tails
Advances over the past year in the misnamed field of “artificial intelligence” have activated the inverse form of the heuristic that haunts so many disabled humans: most people see the language fluency exhibited by large language models (LLMs) like ChatGPT and erroneously assume that the computer possesses intelligent comprehension — that the program understands both what users say to it, and what it replies.
Some critics have attributed this fallacy to pareidolia — the tendency of the human brain to perceive patterns (often humanlike ones, such as faces or voices) in random visual or audio data. But I think that explanation is inadequate, because this phenomenon is demonstrably specific to the use of language. For example, text-to-image generators like DALL-E and Midjourney utilize the same underlying LLM technology as chatbots, but the public release of these programs last year — controversial as they were for other reasons — did not spark the same kinds of zealous speculation about a mind in the machine. Humans don’t have a history of using “image generation” or even “visual art” as a heuristic for intelligence. Only fluency of language has that distinction.
As computational linguist Dr. Emily Bender has said, “We now have machines that can mindlessly generate words, but we haven’t learned how to stop imagining a mind behind them.”
The pop-culture concept of sentient computers — either the ‘fun personal companion’ or ‘scary world-destroying monster’ versions — is blue-sky dreaming with zero real science behind it.
“Languages are symbolic systems,” explains Bender, comprised of “pairs of form and meaning. Something trained only on form” — as all LLMs are, by definition — “is only going to get form; it’s not going to get meaning. It’s not going to get to understanding.”
How does one get to natural language understanding? No one so far has any real idea. It’s considered an ‘AI-complete’ problem, something that would require computers that are as fully complex as, and functionally equivalent to, human beings.
(Which about five minutes ago was precisely what the term ‘artificial intelligence’ meant, but since tech companies managed to dumb down and rebrand ‘AI’ to mean “anything utilizing a machine-learning algorithm”, the resulting terminology vacuum necessitated a new coinage, so now we have to call machine cognition of human-level complexity ‘AGI’, for ‘artificial general intelligence’.)
While the human brain may have been the inspiration for the term ‘neural network’, the neural networks we’ve actually built don’t even properly mimic a single type of brain cell, much less an entire human brain. They don’t do any of the things real brains do; instead they’re mathematically finding statistical patterns in vast quantities of data, a thing that organic brains don’t do well at all. Even if we fully understood how brains work, which we very much do not, simulating a biological nervous system would be infeasibly complex.
Furthermore, it’s not just the brain we’d have to model; emotions are central to the most basic functioning of our brains, and the ability to experience emotions is inextricably bound up with having a body.
As neuroscience professor Giandomenico Iannetti explains, “Our brain inhabits a body that can move to explore the sensory environment necessary for consciousness,” whereas an LLM generates output “by emulating a nervous system but without attempting to simulate it. This precludes the possibility that it is conscious.”
Just to be completely clear: the pop-culture concept of sentient computers — either the ‘fun personal companion’ or ‘scary world-destroying monster’ versions — is blue-sky dreaming with zero real science behind it. Not only are we not close to developing “artificial general intelligence”, we are not even far away from developing AGI, because we haven’t even found a path that could conceivably lead to AGI.
No matter how many ‘artificial neurons’ programmers stack up, or how large a data training set they offer, no LLM transformer can ever possibly develop the faintest comprehension of the meaning behind the words it produces. Any words it spits out about its “feelings” are just fancy Mad Libs. There is no mind, no self-awareness, no sentience, no sapience, no comprehension, no subjective experience or independent agency at work in an LLM, and there never will be. The autocomplete may get spicier, but it’s still just autocomplete.
Follow the money →
None of these well-documented facts have prevented a whole lot of educated people, whom I will graciously assume are intelligent humans in their own right, from falling into the heuristic trap. You’ve probably heard of the Google data scientist who, after hours spent talking with chatbots based on the LLM LaMDA, went public with his assertion that it was “sentient” and “a person”. And the New York Times journalist who had an extended conversation with ChatGPT that he described afterward as “creepy”, “obsessive”, and “stalkery”.
There are many, many more. In the wake of the Chatbot Invasion, every social media platform and media outlet I visit seems to have a tenacious contingent of people breathlessly arguing for the true intelligence of LLMs — either already existing, or just around the technological corner.
Not coincidentally, you’ll often find that the people making the loudest argument for impending computer sentience are literally invested in OpenAI and Google and Microsoft and Meta. I am hardly the first person to point this out, but trillions of dollars are at stake, and these (already extremely wealthy) people reap direct financial benefits when the public mistakes the software’s fluency of language for intelligence and comprehension.
That tech corporations have successfully rebranded the idea of “artificial intelligence” among the media and public to include LLMs is just the tip of the spear. Whenever an LLM says something contrafactual, the companies claim — and the media dutifully reports — that the program is “hallucinating” … a word that suggests both a conscious experience and external senses, neither of which pertain to computer algorithms. Words like “fabricating” and “lying” and even “bullshitting” aren’t much better, because those terms all carry connotations of “intent to deceive”, and no LLM program can ever have intent, either malicious or benign.
“These systems haphazardly stitch together words from their training data because they fit in terms of word distribution, and not because there’s any model of the world or any communicative intent or any reasoning,” says Bender. “It’s not even lying, because lying entails some relationship to the truth that just isn’t there.”
One thing that particularly seems to lead people astray is the way that ChatGPT gives the impression of “apologizing” in response to exterior challenges. OpenAI’s claim that ChatGPT will “admit its mistakes” is worded to suggest that the algorithm both understands that it has made an error and is in the active process of improving its understanding based on the dialogue in progress.
But that’s not at all what’s happening. While ChatGPT does possess a type of very limited continuity among subsequent exchanges in a single “dialogue”, in the sense that it treats the entire session as one long input prompt, there’s zero real-time learning taking place.
There is no mental model of reality for it to update.
I keep seeing screen-shotted human-LLM interactions in which the participating human (and subsequent observers) are so distracted by the “apology” that they fail to recognize that each answer is effectively a dice roll, statistically no more accurate than the previous one. In fact, if you respond to a factually correct answer with a challenge, ChatGPT will usually apologize and give you a different, incorrect answer. You’re basically just shaking the Magic 8 Ball over again until (if you haven’t gotten frustrated and quit first) it randomly serves up an answer you like, at which point you stop asking. But the computer hasn’t “learned” a damned thing from the interaction, any more than the cosmos is sending you a message about the future.
Companies are deliberately guiding these algorithms to emulate a knowledgeable, intelligent, and friendly human, even though the software is exactly zero of those four things.
Making chatbots that seem to apologize is a choice. Giving them cartoon-human avatars and offering up “Hello! How can I help you today?” instead of a blank input box: choices. Making chatbots that talk about their nonexistent “feelings” and pepper their responses with facial emojis is another choice. Making algorithms that “chat” at all — framing every interaction as a “dialogue” — is the most basic choice, and one that we really ought to be side-eyeing more. OpenAI, Microsoft, Google, and other companies are deliberately guiding these algorithms to emulate a knowledgeable, intelligent, and friendly human, even though the software is exactly zero of those four things. Only when this approach gets them into trouble do they backpedal.
One clear reason is that corporations would prefer to use machines for a number of jobs that currently require actual humans who are knowledgeable, intelligent, and friendly, but who also have this annoying tendency to want to be paid enough money to support the maintenance of their inconvenient meat sacks. Not to mention the problematic fact that humans occasionally possess ethics, independent thinking, and objectives besides maximizing shareholder profit.
The second reason for this choice is that, as Ted Chiang wryly notes, “there is a market opportunity for volleyballs.”
The reference is to the movie Castaway, in which a marooned and desperately lonely Tom Hanks makes a washed-ashore volleyball into an imaginary friend. Humans are a social species down to our core; the more modern life erodes our opportunities for actual human companionship — whether it’s by interposing technology as an intermediary into every interaction, or sucking up all our time with the capitalist/consumerist grind — the more desperate we’ll become for friendly-sounding volleyball substitutes.
Profit is also the most likely reason why so many LLM investors are insisting that “AI” represents An Impending Existential Danger to the Human Species! — first, so the public will believe their LLM-based products are more advanced (and therefore worth more money) than they really are, and second, to distract from the very real and significant harms that their use of these programs are already causing to living people today. Computer scientist Timnit Gebru (famously fired from Google in 2020 for raising ethical issues around the use of AI) has repeatedly warned that when the public conversation focuses on red herrings — like the potential morality and values of a wholly-theoretical computer intelligence — we cease to ascribe responsibility to the real people and actual corporations that are creating harmful products. Any action taken to counteract those harms would cut into profit, so the LLM-invested folks would really prefer we all worry about a mirage instead.
Dueling heuristics spawn chaos and confusion
LLM transformers work by executing complex pattern-matching across a vast data set, with a dollop of mathematical randomness thrown in to make them appear less robotic. This combination makes LLMs astoundingly good at mimicking complex, human-sounding language; ironically the same architecture is also precisely why they are so factually unreliable.
Here’s the paradox: even while the language fluency fools us into imagining these chatbots are ‘persons’, we simultaneously place far more trust in their accuracy than we would with any real human. That’s because the fact that we still know it’s a computer activates another, more modern rule of thumb: that computer-generated information is accurate and trustworthy.
When we were dealing with pocket calculators, that was a valid and time-saving assumption. In the late twentieth century, there was absolutely no need to double-check whether your TI-30 was accurately reporting the square root of seventeen. But LLMs have now shattered the usefulness of the “computers are reliably accurate” heuristic as well, biting everyone from lawyers to university professors in the ass along the way.
“Any human-style explanation is not necessarily correct; we are fallible,” says linguist Noam Chomsky. “But this is part of what it means to think: To be right, it must be possible to be wrong. Intelligence consists not only of creative conjectures but also of creative criticism. Human-style thought is based on possible explanations and error correction, a process that gradually limits what possibilities can be rationally considered. ChatGPT and similar programs are, by design, unlimited in what they can “learn” (which is to say, memorize); they are incapable of distinguishing the possible from the impossible.”
Corporations would like for us to think of LLMs as both humanlike and accurate, but those two ideas are inherently in conflict. You can’t have it both ways.
As a society, we’re going to have to radically rethink when and how and even if it makes sense to trust any information that either originates from, or is mediated by, any kind of machine-learning algorithm — which, if you think about it, currently encompasses nearly All The Things. This is a giant clusterfuck of a paradigm shift that’s only in its earliest stages; the upheavals will keep coming, fast and furious, and all I can say with certainty about that is: buckle up kiddos, it’s gonna be a ride.
By pointing out that these are two sides of the same flawed coin, I hope to encourage more people to question their automatic assumptions about language and intelligence in disabled humans as well.
Much more quickly — I would guess over the next year or so — I predict the broader public will be forced to decouple language fluency from intelligence assessment, at least when it comes to overestimating chatbots and other algorithmic software … if only because the messy, embarrassing failures will proliferate until most of us learn to stop equating the two.
That doesn’t mean, however, that people won’t still default to using the heuristic in ways that underestimate the intelligence of real live human beings. But by pointing out that these are opposite sides of the same flawed coin, I hope to encourage more people to question their automatic assumptions about language and intelligence in disabled humans as well.
Relatedly, I wrote this story …
The dyspraxic, autistic narrator of my short story “Hope Is the Thing With Feathers” befriends a genetically engineered corvid in their care at a university animal-study lab. But before that, they come to feel a kind of kinship with even the ordinary, wild-type crows:
They’re smarter than most people realize, which is something I can relate to. Corvids can learn to speak human languages, but they rarely use more than a few words, whereas some African Greys can learn hundreds, and speak in whole phrases. Because they don’t talk, everyone — even the researchers, often — assumes crows aren’t as intelligent as parrots. But that’s not true.I couldn’t speak at all until I was ten years old, and even when I did start talking, it wasn’t in a ‘normal’ way. Everybody — doctors, teachers, even my own parents — assumed I was intellectually disabled. They either talked down to me like I was a pet, or around me like I didn’t exist.
I understood everything, though. I mean, I have an auditory processing disorder, so I can’t parse a long series of instructions, or sort out sounds in a noisy environment — but I was listening the whole time.
Even when people were discussing how stupid I was.
If you’d like to read the rest of “Hope”, it appears in the July/August 2023 issue of Asimov’s Science Fiction magazine, in most bookstores and on newsstands in the US and UK, now through early August. In the UK, you can also buy a single print copy online from Newsstand through July 11th. The digital version is available for online purchase from Magzter.
Researching, composing, and sourcing informative essays like this one require many, many hours of unpaid work. If you appreciate it and can afford to, please help me continue writing with a Ko-fi tip or a paid Substack subscription. Thank you!
Footnotes
Allistic is the word coined by the Autistic community to refer to ‘a person who is not autistic’; it is more specific and accurate in this context than ‘neurotypical’.
Here’s a tip: if you’re researching autism, and you see a study co-authored by Simon Baron-Cohen, or an article that quotes him uncritically as an expert, stop reading immediately, print it out, and set it on fire. Preferably while screaming at the top of your lungs.
Autistic people tend to prefer the term ‘nonspeaking’ over ‘nonverbal’, due to linguistic accuracy. Apraxics don’t have speech, but many of them do — or could — have words and/or language.
I have capital-I Issues with the whole concept of ‘intelligence’, as well as the measurement thereof, but they lie beyond the scope of this essay. I did touch on some of them in “Elon and Me”, though.
Yes, this also precludes any kind of ‘uploaded consciousness’ — without our bodies, we don’t have recognizable minds. (I’m still a big fan of the satirical Amazon series “Upload”, however.)
Increasingly I find myself wanting to take journalists by the shoulders and shake them until their teeth rattle. Like Jeff Jarvis, who first correctly noted that ChatGPT “is a personal propaganda machine that strings together words to satisfy the ear, with no expectation that it is right” … and then turned around in the same article [paywalled link] and discussed his “anthropomorphic sympathy” for ChatGPT as a “wronged party”. FOR FUCK’S SAKE, JEFF!
This should be obvious if you stop and think about it for a half a second — what corporation is going to let the inputs of random members of the shitposting public train its very expensive algorithm on the fly?
On the other hand, of course anything you type into a chatbot will be saved and may be used by the owner company as training data in the future.
When I was writing for a broad audience of strangers in my twenties, it never occurred to me to worry about whether readers would recognize my references. Now that adults thirty years younger than me exist, I experience a constant simmering uncertainty about the obscurity of my analogies. Do Gen Z kids know what Mad Libs are, or a Magic 8 Ball? I have no idea! Here, have a Wikipedia link.
I’ve used an archive link, figuring the number of non-millionaires outside the finance industry who have a Financial Times subscription is small. If you prefer, you can see the original article.
I’ll note here that Emily Bender explicitly does not wish her now-famous phrase ‘stochastic parrots’ to cast doubt on the relative intelligence or internal experience of actual parrots — the idea, she said, was to “give a cute metaphor that would allow people to get a better sense of what this technology is doing. It’s honestly unfair to parrots. I like to say that we’re drawing here really on the English verb ‘to parrot’, which is ‘to repeat back without any understanding’, and remaining agnostic about the extent to which parrots have internal lives and know what’s going to happen when they say certain things.”
My own decades of reading on the science of animal behavior and consciousness have led me to the conclusion that many species of animal, including birds like parrots and corvids, fall much closer to humans in both sentience and intelligence than most people are comfortable with acknowledging. But that, perhaps, is another whole essay.