What are examples of thought experiments against or in favour of strong AI, apart from the Chinese room argument?

Question

The Chinese Room argument against strong AI overlooks the fact that "the man in the room" is acting as a macro-scale "neurotransmitter" of the larger system in which he resides. It does not rule out strong AI, it simply reduces to an enigmatic question: where does understanding "reside" and how does it epiphenomenally emerge?

What are other examples of thought experiments against or in favor of strong AI (apart from the Chinese room argument) or extensions or refutations to known experiments?

Answer 1

An excellent book summarizing the development of thought in this area over several hundred years is Mind Design II, edited by John Haugeland. This book contains a collection of essays written by the major thinkers in this area through until about the 1990s. A brief summary of some of the major ideas is:

1. Descartes: Minds are spirits, brains are bodies. Minds attach to brains in a special part of the brain. Descartes has a number of thought experiments, but they mostly have a Theological root. Most modern scientists do not find these to be satisfying because the resolve metaphysical questions about the mind by assuming you already believe in a particular interpretation of a God.
2. Gilbert Ryle argues in the late 1940s that concepts like the Mind are in some sense vacuous. Asking whether a machine has a mind or not is like asking whether a field contains two cows or a pair of cows: this is a conceptual distinction that has no relation to the real world. Because of this, its study is futile. The parallel movement of Behaviorism in psychology also viewed the study of minds and mental states as a fool's errand. Rather, this movement focused on studying behaviors that could be directly measured.

3. Turing publishes his seminal paper in AI, in 1950. In this paper, Turing anticipates nearly all later objections to AI, and summarizes reasonable refutations of them. This paper contains a number of short thought experiments. One of the more notable ones is the Turing Test. A key observation Turing makes is that if a machine behaves in a way indistinguishable from a human, then any arguments used to reject its claim to intelligence (and even to consciousness) appear to work equally well on other humans. Searle doesn't really address this in his later argument. Basically, I have no way to know for sure whether you have subjective experiences of the same kind I have. The fact that you're made of roughly the same kind of meat as I am seems like a shaky explanation. For example, would humanoid aliens that act like us also be intelligent? What if it were found that their brains took some radically different form from ours? What if it were found that their brains were really networks of transistor-like machines?

4. The Cognativist revolution spanned a number of fields starting in the 1960s. This movement held that behaviorism was wrong for two basic reasons. First, Behaviorism does not account for the subjective feelings of existence or understanding. Second, Behaviorism could not explain phenomena like language, that appeared to involve reasoning logically about symbols. Notable authors in this period are Chomsky and Fodor, who argued that minds were essentially computer programs that happened to be running on brains instead of computers. These ideas dominated AI, Psychology and Linguistics for about 30-40 years. Fodor's works contain a number of thought experiments involving language and programs.

5. John Searle's Chinese Room argument caused problems for the cognativists, because it presents a convincing chain of logic showing that a program that does complicated tasks (like linguistics) does not subjectively understand what it is doing. Various objections are raised and Searle responds to them in his publication of the argument. Objections to Searle's arguments form the basis of several later movements in philosophy of AI.
6. The Connectionists argue that we can represent the behavior of the brain (and thus, the mind) with a brain-like program. This paralleled increasing interest in artificial neural networks. An example of a cognativist thought experiment is to imagine a simulation of an entire human brain, conducted with perfect fidelity (for instance, we accurately model all the quantum mechanical interactions). If such a machine were not conscious, opponents need to answer the question of where consciousness is hiding (Is it in the meat? Then see Turing. Is it in some yet-undiscovered property of neurons? Then the burden of proof is definitely on the person proposing such a hidden property, and they must explain why we cannot simulate it).
7. The Churchlands and others argued, much as Ryle, that minds as the public understand them are a sort of "folk theory", that appear to explain something about reality to lay people, but are actually a dead end. Concepts like "beliefs" and "experiences" are, according to these arguments, akin to aether or phlogiston: things that our current, malformed, understandings of neuroscience seem to require, but that in fact have no basis in reality, even though nearly everyone believes in them at present. This line of work contains many extensions of the kinds of thought experiments Turing hinted at.
8. Brooks and others advance the theory of embodied cognition, in which the mind is not confined to the body, but is actually a property of a body plus the environment it is placed in. An example thought experiment from this school (by Andy Clarke if I recall correctly) is to imagine a person with profound loss of the ability to form new long term memories. This person carries around a notebook. The cover says "Your memory". The first page contains instructions explaining that the person has lost the ability to remember things, and that they write new things they want to remember in the book. The person can remember things for, say, 15 minutes at a time, and the book is very well organized. Where is this person's mind exactly? Is it in their brain (which does not remember anything 15 minutes in the past), or is it in the brain and the book, which together can remember and reason about things essentially like a normal person. If it is in the brain and book, what if we replace the book with a computer inside the person's head?