# Is the "Chinese room" an explanation of how ChatGPT works?

Sorry if this question makes no sense. I'm a software developer but know very little about AI.

Quite a while ago, I read about the Chinese room, and the person inside who has had a lot of training/instructions how to combine symbols, and, as a result, is very good at combining symbols in a "correct" way, for whatever definition of correct. I said "training/instructions" because, for the purpose of this question, it doesn't really make a difference if the "knowledge" was acquired by parsing many many examples and getting a "feeling" for what's right and what's wrong (AI/learning), or by a very detailed set of instructions (algorithmic).

So, the person responds with perfectly reasonable sentences, without ever understanding Chinese, or the content of its input.

Now, as far as I understand ChatGPT (and I might be completely wrong here), that's exactly what ChatGPT does. It has been trained on a huge corpus of text, and thus has a very good feeling which words go together well and which don't, and, given a sentence, what's the most likely continuation of this sentence. But that doesn't really mean it understands the content of the sentence, it only knows how to chose words based on what it has seen. And because it doesn't really understand any content, it mostly gives answers that are correct, but sometimes it's completely off because it "doesn't really understand Chinese" and doesn't know what it's talking about.

So, my question: is this "juggling of Chinese symbols without understanding their meaning" an adequate explanation of how ChatGPT works, and if not, where's the difference? And if yes, how far is AI from models that can actually understand (for some definition of "understand") textual content?

• You are already anthropomorphizing when you asay it has a "good feeling"! Feb 25 at 22:03
• The distinction you make between algorithmic and AI/learning does not exist. You might view AI/learning as algorithmic as well, having a very detailed set of instructions. Too detailed for any (group of) humans to type up. The finer details of the algorithm are determined algorithmically, by training on a large dataset. There is certainly no need to invoke the concept of 'feeling'.
– user68640
Feb 26 at 12:16
• To expand; it is not clear how you intend to apply the concepts of 'understanding' or 'knowing' to ChatGPT. It simply 'does'.
– user68640
Feb 26 at 12:17
• Considering how poor ChatGPT is at counting (e.g. stating "Here are 5 examples why:" and listing anywhere between 3-7 examples) and graphical AI in general at drawing hands (counting fingers is hard), there's definitely some degree of "without understanding their meaning" indeed.
– Mast
Feb 27 at 8:45
• We don’t have a concrete model of intelligence. We don’t have a mathematical theory of intelligence. All we have are some thought experiments and ill-defined philosophical concepts. Historically, when humans try to build models of the real world based on their amazing deductive powers without evidence, they come up with stuff like the world being made of someone’s armpit or, at best, Lamarckian evolution. Feb 28 at 13:37

Yes, the Chinese Room argument by John Searle essentially demonstrates that at the very least it is hard to locate intelligence in a system based on its inputs and outputs. And the ChatGPT system is built very much as a machine for manipulating symbols according to opaque rules, without any grounding provided for what those symbols mean.

The large language models are trained without ever getting to see, touch, or get any experience reference for any of their language components, other than yet more written language. It is much like trying to learn the meaning of a word by looking up its dictionary definition and finding that composed of other words that you don't know the meaning of, recursively without any way of resolving it. If you possessed such a dictionary and no knowledge of the words defined, you would still be able to repeat those definitions, and if they were received by someone who did understand some of the words, the result would look like reasoning and "understanding". But this understanding is not yours, you are simply able to retrieve it on demand from where someone else stored it.

This is also related to the symbol grounding problem in cognitive science.

It is possible to argue that pragmatically the "intelligence" shown by the overall system is still real and resides somehow in the rules of how to manipulate the symbols. This argument and other similar ones try to side-step or dismiss some proposed hard problems in AI - for instance, by focusing on behaviour of the whole system and not trying to address the currently impossible task of asking whether any system has subjective experience. This is beyond the scope of this answer (and not really what the question is about), but it is worth noting that The Chinese Room argument has some criticism, and is not the only way to think about issues with AI systems based on language and symbols.

I would agree with you that the latest language models, and ChatGPT are good example models of the The Chinese Room made real. The room part that is, there is no pretend human in the middle, but actually that's not hugely important - the role of the human in the Chinese room is to demonstrate that from the perspective of an entity inside the room processing a database of rules, nothing need to possess any understanding or subjective experience that is relevant to the text. Now that next-symbol predictors (which all Large Language Models are to date) are demonstrating quite sophisticated, even surprising behaviour, it may lead to some better insights into the role that symbol-to-symbol references can take in more generally intelligent systems.

Yes it is a good analogy, as explained nicely by Neil.

how far is AI from models that can actually understand (for some definition of "understand") textual content?

Here's the catch: how do we know that we (humans) are not simply very sophisticated chinese rooms?

For instance suppose that current AI models are improved so much that their performance is on par to human performance, without the current catastrophic failures, yet they retain the current model architectures. Now you have an apparent paradox: they are indistinguishible from humans and yet you know that they are not "understanding".

Personal guess: It's chinese rooms all the way down.

• Isn't this also what Turing was trying to demonstrate with his Imitation Game? Feb 25 at 23:41
• That seems like a pretty big "if". It will probably never be possible for purely language-based models to, for example, produce a correct original proof of an unproven math theorem. Feb 25 at 23:41
• @Barmar Not sure about Turing intent, but my understanding is that he considered the Turing test as a definition of intelligence. If you pass it, you are intelligent. Today it is widely considered unsatisfactory, due to many techniques to fool the judges. For instance the AI may play ignorant and forgetful, which is very "human". Feb 26 at 0:32
• Regarding your final personal guess, it cannot be an infinite regress like turtles all the way down since human's neural system is physical and thus limited and finite in terms of levels of such rule-based symbol manipulations which is supposedly to account for the distinct "hard qualitative" understanding of human beings. Feb 27 at 19:20
• @Rexcirus Note that AlphaGo and its chess equivalents are not a language-based models, but traditional tree search engine augmented by neural networks to help determine where to search. A similar approach could possibly be applied to prove theorems. Feb 28 at 10:22

Searle's Chinese room is not intended as a functional description of any real-world machine. Searle was a philosopher who created the Chinese room as a thought experiment to show what he considered an absurd conclusion of the computational theory of mind. The intended absurdity is that the person inside the room doesn't understand anything about the inputs or outputs, but that when just looking at the room from the outside, the room (i.e. the system person+dictionary) appears to understand Chinese.

To Searle, it was clear that there was no "understanding" located anywhere here, and so this system was clearly not equivalent to a human consciousness that actually understands Chinese. But "strong AI" computationalists believe that all that matters for consciousness is the inputs and the outputs. Since Searle considers the conclusion that the room is conscious absurd, this thought experiment is supposed to be a refutation of this computationalist viewpoint of consciousness.

ChatGPT and other large language models are not a realization of Chinese rooms. There isn't a human in there who doesn't understand English and instead uses a dictionary or set of rules to translate inputs to outputs.

The point of the Chinese room is that it is clear that a) the human doesn't understand Chinese and b) the dictionary/rules are just a book that isn't conscious in itself either, otherwise it doesn't work as a reductio ad absurdum. The point of the thought experiment is that it eliminates anything to which we could attribute understanding - but indeed one of the replies to Searle was that it was just the room itself that had understanding/consciousness, and the interplay between the human and the dictionary is just analogous to the way different regions of the human brain might interact to produce the overall "understanding".

Instead, large language models consist of a big neural network that transforms the inputs to outputs. It's not two distinct entities like in the Chinese room - "rules storage", i.e. the dictionary, and "rules implementor", i.e. the human - it's one big algorithmic structure whose exact inner workings are often hard to explain for specific use cases. You may or may not assign the ability to "understand" to this network, but there are no identifiable substructures here as there are in Searle's room.

These models, of course, raise much the same questions about consciousness and understanding that Searle's Chinese room does, but the room with its clear two-component structure bears no actual resemblance to how the transformer networks underlying large language models work. You might argue that these models are not conscious in the same way that Searle's room is not conscious, but your argument for why they aren't conscious (or why they don't "understand" the language they're using) needs to be very different from Searle's argument.

• If the person can respond with perfectly reasonable sentences, they clearly do have a fairly good understanding of the relation between Chinese symbols. They just don't have an understanding of the relation between those symbols and the real world. It seems far from trivial to conclude that one of those involves understanding and the other one doesn't. If someone spends enough time in a Chinese room, and you then tell them what a few of those symbols mean (if they don't figure it out themselves), that may very well make everything fall into place to have them perfectly understand the language. Feb 27 at 15:40
• @NotThatGuy Note that I am careful to say that Searle says it is clear that there's no "understanding" here, not that this is an indisputable fact. The "Replies" section of the Wiki article I link in the first sentence has plenty of viewpoints that don't accept the Chinese room as an argument against computationalism. Feb 27 at 16:27
• ChatGPT et al. are precisely implementations of Chinese rooms. Symbols go in, rules are applied, symbols go out, where is the understanding? The fact the rules are applied by a high-speed machine and not a human seems irrelevant. Feb 28 at 10:23
• @user253751 That's not a fact, that's your judgement: You are already certain that ChatGPT has no understanding, so you say it's just an implementation of a Chinese room. But a computationalist would say a human is also just a Chinese room: Words go in, a neural network (the brain) processes them, words come out. But e.g. Searle would disagree with this statement. That is, "X is an implementation of a Chinese room" is not a factual description of X, it's based in certain philosophical positions on what it means to be conscious and which things are or are not conscious. Feb 28 at 10:57
• @ACuriousMind Notice that I did not say anything about humans but only about ChatGPT. Feb 28 at 12:39

The chinese room argument is useless because it can be applied to the brain as well. Replace the slit with sensory input, the handbook with the wiring of the brain and the activity of the agent inside the room with neuron activity. In the same way the argument demonstrates that the room has no understanding it demonstrates that the brain has no understanding.

My personal assessment is that LLMs like chatGPT have a true understanding of the domain they were trained on. My reasoning is that the training forced the model to squeeze all the information it can utilize to make its predictions into the limited amount of its network parameters. In this regard the incorporation of understanding is a far more efficient usage of the available space than any other kind of data compression.

As many have stated, the Chinese room analogy is intended to show that any hardware + software instance that relies on rules alone (logical operators on input symbols) cannot be said to have understanding. It is not a good analogy, and the argument does not apply well to trained neural networks. Neural networks (ChatGPT is actually two trained NNs -- an input-trained NN and output-trained NN) are produced as a result of extensive training (for ChatGPT, unsupervised training to generate a language model, and a couple of stages of supervised training on its output sentences). This training creates many billions of weights that are instantiated in the NN, and these weights are applied across the NN nodes as it 'processes' an input prompt. From a macro view, the NN code is written in the logic of computer language, so one might conflate this with the logical operations on input signals described in the Chinese Room analogy, concluding that a NN is nothing but logical operations. This is a mistake, in my view. A trained NN is different in kind from the purely operational program that Searle described. By evolving weightings through training, NNs encode an incredibly complex object that Searle could not have imagined when he created the analogy. Does a neural network then have some kind of understanding? I think that it's clearly not anything like human understanding, but I also believe that it is at least some kind of understanding. There are many aspects to this discussion, to say the least. One main objection to Searle's argument is that if you look at individual neurons in the brain, you will not find understanding there either, but the physical brain does give rise to consciousness in some way (unless you believe in some extra 'magic' that overlays the physical brain -- something that Searle, a physicalist, would not allow).

I suggest it makes all the difference in the world whether 'the knowledge' was acquired by parsing many examples to get 'a feeling for what's right' (AI/learning), or by detailed instructions (algorithmic).

Can you say how the exposition here relates to the Question title?

What point is there but that the AI system should and the algorithmic should not be able to expand its programmed capabilities?

How could anyone's understanding Chinese matter?

Do you know - or know of - anyone who believes ChatGPT understands anything? More importantly, anyone who can explain where most people's understanding comes from?

Do you know of people who know how to choose words based on anything but what they've seen?

Can you look again at '…it mostly gives answers that are correct?

Is that less or more useful/worth-while than the answers most people give?

How could understanding Chinese matter, unless you were specifically asking for translations? Are you?

I suggest that juggling Chinese symbols without understanding their meaning is irrelevant to ChatGPT.

If what you're really Asking is how far AI might be from 'understanding' anything, why not first define 'understanding'?

• Do you know - or know of - anyone who believes ChatGPT understands anything? Yes - people who don't understand how current AI works, i.e. most of the planet. Feb 28 at 10:40
• @IanKemp Ho, ho, ho. Was that pure cynicism or was there a useful point hidden in there? If you meant 'most of the planet' might believe that, had they any understanding or had ever heard of ChatGPT or anything similar, please Post some details. Mar 1 at 0:25
• @IanKemp People who know a whole lot about AI, and some about neuroscience, might also wonder about the difference between those things, and whether AI can really be said to not understand things. Mar 1 at 11:53