34
$\begingroup$

After seeing StackOverflow's banning of ChatGPT, I explored it out of curiosity. It's marvellous as it can write code by itself!

Later to check if it knows chess as well like Google-Deepmind's AlphaZero AI, I asked below questions:

Me: Does openai know how to play chess?
OpenAI: It is not clear whether OpenAI knows how to play chess. As an artificial intelligence research organization, OpenAI has developed a number of advanced machine learning algorithms and techniques that allow AI systems to learn and adapt to new tasks and situations. ...

Me: In a chess if white has a king & bishop and if black has only a king, then who can win with a perfect play?
OpenAI: In a chess game where white has a king and a bishop and black has only a king, it is possible for either player to win with perfect play. This type of endgame is known as a king and bishop versus king endgame, and it is considered one of the most challenging in chess. ...

Now this answer is entirely wrong. Because as soon as only a bishop (or a knight) remains apart from the kings, it's instantly declared as a draw!

Question: Shouldn't the AI reject my question on a subject which is not in its known criteria?
It does so for many other subjects.

Note: Should we replace bishop = rook then ChatGPT answers exactly the same answer with replacing those pieces. However that happens to be true.

Improve this question
$\endgroup$
5
  • 34
    $\begingroup$ I think a good way to think about ChatGPT is that it doesn't know anything about anything, and its goal is to bluff its way through a conversation by guessing what you want to hear. Sometimes, the bluff is easier to call. $\endgroup$
    – Stef
    Commented Dec 13, 2022 at 10:41
  • 3
    $\begingroup$ "that happens to be true" (in the rook case) - really? The GPT answer appears to be saying that either player can win; I'm not a chess expert but I'm struggling to see how black can force a win with just a king? $\endgroup$
    – psmears
    Commented Dec 13, 2022 at 15:57
  • $\begingroup$ @psmears, sorry.. A small correction here is that, if i replace "bishop" with "rook" then the chatgpt correctly says that white can win. No mention of Black's win. $\endgroup$
    – iammilind
    Commented Dec 13, 2022 at 18:11
  • 1
    $\begingroup$ Related: The Chinese room argument $\endgroup$
    – Ouroborus
    Commented Dec 13, 2022 at 18:45
  • 2
    $\begingroup$ "that happens to be true" you might want to experiment by retrying the same prompt multiple times and observe that the response may change or even contradict previous response. $\endgroup$
    – Andrew T.
    Commented Dec 14, 2022 at 4:37

6 Answers 6

Reset to default
72
$\begingroup$

ChatGPT is a large language model. That means it's very good at stringing together words in ways that humans tend to use them. It's able to construct sentences that are grammatically correct and sound natural, for the most part, because it's been trained on language.

Because it's good at stringing together words, it's able to take your prompt and generate words in a grammatically correct way that's similar to what it's seen before. But that's all that it's doing: generating words and making sure it sounds natural. It doesn't have any built-in fact checking capabilities, and the manual limitations that OpenAI placed can be fairly easily worked around. Someone in the OpenAI Discord server a few days ago shared a screenshot of the question "What mammal lays the largest eggs?" ChatGPT confidently declared that the elephant lays the largest eggs of any mammal.

While much of the information that ChatGPT was trained on is accurate, always keep in mind that it's just stringing together words with no way to check if what it's saying is accurate. Its sources may have been accurate, but just writing in the style of your sources doesn't mean that the results will themselves be true.

Improve this answer
$\endgroup$
10
  • 11
    $\begingroup$ Exactly. It finds the words that are most likely to follow the prompt, and has no understanding of anything else. $\endgroup$
    – Oliver Mason
    Commented Dec 12, 2022 at 12:47
  • 3
    $\begingroup$ I know that on previous GPT models, these kinds of questions with no right answer could lead to some amusing answers. More like this: Baltimore Orioles effect and Botsplaining and Galactica: the AI knowledge base that makes stuff up and when humans do it Comparative Illusion $\endgroup$
    – Criticizing Israel not allowed
    Commented Dec 12, 2022 at 16:32
  • 26
    $\begingroup$ It's fairly easy to confuse ChatGPT once you understand this principle. For example, if you ask it to describe the Avatar movies it'll confuse James Cameron's Avatar with Avatar: The Last Airbender. Or if you ask it to describe how the planets follow probabilistic orbitals it'll happily confuse Quantum Mechanics and Newtonian gravity for you. All it takes is two concepts that use the same set of words, because the language model has no deeper understanding. $\endgroup$
    – David
    Commented Dec 12, 2022 at 22:40
  • 1
    $\begingroup$ Moreover, you claim "making sure it sounds natural". Right now, I am not familiar with the details of the ChatGPT, but I will soon be, but does it really ensure it sounds natural? If it's a statistical model with no constraints on "naturalness", then I doubt that claim is true. $\endgroup$
    – nbro
    Commented Dec 14, 2022 at 12:36
  • 3
    $\begingroup$ This is why it's less impressive than it looks. Still impressive, sure, but more showy than anything else. $\endgroup$
    – OpenAI was the last straw
    Commented Dec 14, 2022 at 13:50
20
$\begingroup$

ChatGPT does not actually know anything. But more importantly even, it does not know this fact! Hence, it does not know that it does not know.

It is only good at combining text.

Improve this answer
$\endgroup$
5
  • 5
    $\begingroup$ Note that this is just a philosophical point of view. Who decides that people also know anything? What do you mean by "knowing"? Maybe we also don't know anything. We have memories, which we recall with some algorithm, based on some context. You need to define "to know". Because, if "to know" means "have knowledge of something" and "knowledge" means "info or facts", then ChatGPT certainly knows something, because it can give you some facts. Either it's aware of what it says, it's another question, which also depends on what you mean by "awareness". $\endgroup$
    – nbro
    Commented Dec 14, 2022 at 12:38
  • 1
    $\begingroup$ So, what one can say for sure is that ChatGPT may give you some facts and correct info, but these facts are just the "most likely" outputs given the data and the inductive bias, i.e. it's a statistical model, and thus it's limited by how the model is defined, the data and the training algorithm. So, it can also give you completely wrong info. The way it produces some output $x$ is, we can say, not equal to how we produce the same output and, most likely, not even close to how we produce it. $\endgroup$
    – nbro
    Commented Dec 14, 2022 at 12:48
  • 4
    $\begingroup$ There is a great article describing this phenomena (which it calls "pseudo-profound bullshit") when done by humans. Cult leaders and snake-oil salesmen are great at stringing together buzzwords into grammatically correct but meaningless sentences, which the unaware listener might think to hold some deep knowledge. $\endgroup$
    – vsz
    Commented Dec 15, 2022 at 19:09
  • 3
    $\begingroup$ @nbro the aspect of awareness is relevant, I agree. One way to naively put it, is when there is knowledge/awareness regarding the fact, that a specific knowledge is not available. Humans, in the best case, are aware of their individual lack of knowledge. We have some sort of table of content about things we know. We also know that we may have heard something, but never thought it through ourselves. ChatGPT and other AI models store and recombine/associate. $\endgroup$
    – DrCommando
    Commented Dec 15, 2022 at 20:58
  • $\begingroup$ So interesting comments. Knowledge about facts, and knowledge about how to check new facts against our knowledge base. How to expand our table of contents through valid reasoning. $\endgroup$
    – Jaume Oliver Lafont
    Commented Dec 16, 2022 at 8:59
15
$\begingroup$

ChatGPT and other GPT-based machine learning models don't actually know anything in the sense you're thinking of. ChatGPT is a distant descendant of Markov chain text generators such as Dissociated Press, and works by predicting what is most likely to follow a given sequence of text.

So, when you ask it "Does openai know how to play chess?", what you're actually getting is the answer to the question "What is most likely to follow the phrase 'Does openai know how to play chess?'?"

Improve this answer
$\endgroup$
2
  • $\begingroup$ Then how does it write code? $\endgroup$
    – user71244
    Commented Apr 29, 2023 at 16:47
  • $\begingroup$ @Shub, the same way it answers questions. Code isn't fundamentally any different from written language, and ChatGPT tends to make the same sorts of mistakes, producing code that is syntactically correct but semantically nonsensical. $\endgroup$
    – Mark
    Commented Apr 30, 2023 at 4:35
9
$\begingroup$

“It” does nothing. Don’t think just because every conman out there calls our really shockingly primitive neural nets “AI”, and wants to convince you that it’s actually an autonomous intelligence, that it’s not just a glorified function (a list of commands to blindly execute, not a person), to apply a set of biases onto a given input pattern, that have been programmed into it in a way that the programmer doesn’t “have to” know what he’s doing or even what precisely he wants. :)

It is just biasing for the patterns in its training data. And giving you whatever that results in for your your input. In this case, if I am correct, applying its output to its input too again, and again, with diminishing sanity.

So the answer is that your input will just be treated like it is a spectrum of those patterns, no matter what.
In other words: If all it knows is a hammer, everything will look like a nail. :)

So it is quite mundane, and nothing magical at all.

Everything beyond that, attributed to such systems, is deliberate dupery, to get people to “invest” money.

(Don’t get me wrong: This technology is useful when writing an actual algorithm really is beyond a human’s capabilities. (E.g. by definition a brain cannot imagine a model of itself in its entirety. Or you cannot write down the entire set of experiences of a lifeform, find all the patterns, and turn them into a set of rules manually. Even if the Pauli Exclusion Principle would not exist.) But nowadays it is abused by people who proudly can’t even define what they want, to just throw examples at, and expect to get an universal function out, so they can call themselves geniuses and get lots of money.)

Improve this answer
$\endgroup$
21
  • 13
    $\begingroup$ I don't really disagree with any of those statements. However, what else do humans do? Our brains are also just glorified functions that map sensory inputs (plus some nondeterministic crud) to commands to the muscles. If a human tries to answer a question about a subject she doesn't know, she will also make an attempt based on patterns she has seen before that seem to resemble those in this new situation. The main difference with ChatGPT is that it trumpets out wildly extrapolated guesses with the same matter-of-factly confidence, as answers where it really does have good training basis. $\endgroup$
    – leftaroundabout
    Commented Dec 12, 2022 at 19:03
  • 5
    $\begingroup$ @leftaroundabout Your view is an extremely common fallacy about how our brains work. Our brains are not information processors. We actually don't really know how our brains work. $\endgroup$
    – Nelson
    Commented Dec 13, 2022 at 5:27
  • 4
    $\begingroup$ @Nelson I also agree with that we don't understand how our brains work, but this is besides the point. Neither do we understand how deep learning works. Certainly they don't work the same, but at this point it's clear that systems like ChatGPT do behave a lot like, specifically, humans tasked with writing pseudo-profound essays. It's tempting to rest on the pillow of smugness for all the things we can still do better than ChatGPT, and that these are the “truely human” skills, but unfortunately it's already No-True-Scotsman–“true”. $\endgroup$
    – leftaroundabout
    Commented Dec 13, 2022 at 8:19
  • 5
    $\begingroup$ @leftaroundabout If as a human being I was asked this chess question I could separate my thought process into two parts a) check my factual knowledge of chess for an answer and b) formulate this answer into a nice text that other humans can read and understand. ChatGPT appears to do very well on part b) but doesn't do anything related to part a) If my human brain comes up empty on a) I could still apply b) and produce an answer similar to ChatGPT but in general I don't. $\endgroup$
    – quarague
    Commented Dec 13, 2022 at 9:05
  • 7
    $\begingroup$ @Nelson Well, if a computer outputs a perfect word-for-word copy of the poem "The Raven" without using any external sources, we conclude that the computer must have contained a representation of the poem. When a human brain performs the same feat, why would we not come to the same conclusion? $\endgroup$
    – Sophie Swett
    Commented Dec 15, 2022 at 2:20
2
$\begingroup$

The original question asked about "an AI" generally, yet most of the responses here focus on OpenAI's ChatGPT specifically. Seems like the answer would depend on the specific type of AI being used, not limited to just large language model-based chatbots, but considering other types of knowledge representation systems more generally.

Not being an AI expert, I can't speak to this, but here's some good background reading: https://en.wikipedia.org/wiki/Knowledge_representation_and_reasoning

Improve this answer
$\endgroup$
0
$\begingroup$

I agree with most the former answers here, and apologies I don't have high enough points yet to comment so had to do this as an answer, but I believe due to inaccuracies, lack of fact checking capabilities etc... in the data its trained on up to 2021 currently at the time of writing, fine tuning via it's API has become popular I believe. Although you were mainly asking about ChatGPT itself I felt its a good idea to consider it's API as part of that which is just as easily accessible, I am currently learning how to fine tune it my self for more specific and accurate results.

So in respect of the question: Shouldn't the AI reject my question on a subject which is not in its known criteria? It does so for many other subjects.

It can possibly do this if you train / fine tune it another way on the API, but for the latter part, using the normal ChatGPT and ChatGPT plus user interface part of the site, not the API the AI mostly will not reject your question unless it breaches ethical and moral constructs, which I think is also another important aspect to consider when gaining information from ChatGPT.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .