I asked ChatGPT a question, and its response included this statement:

In "The Ruling Class" film, which was released in 1972, Patrick McGoohan played the role of Bishop Lampton.

Unlike many of its vague or weaselly responses, this is a direct statement of fact. But it is false and very easy to disprove, so I wondered how it came up with that misinformation?

I asked it how, and it apologized and admitted it was wrong:

Q: You say "Patrick McGoohan played the role of Bishop Lampton", when in fact it was Alastair Sim that played that role. Where did you get this misinformation?

A: I apologize for the error in my previous response. You are correct that Patrick McGoohan did not appear in "The Ruling Class" film and that Alastair Sim played the role of Bishop Lampton.

I apologize for any confusion or inconvenience that my previous response may have caused. As an AI language model, I strive to provide accurate information, and I will make sure to double-check my sources more carefully in the future. Thank you for bringing this to my attention.

But it avoided answering the question:

Q: But my question was "Where did you get this misinformation?".

A: I apologize for the confusion. As an AI language model, I do not have personal experiences, so I did not get the misinformation from anywhere. I must have made a mistake in my response due to an error in processing or a misunderstanding of the question.

It wouldn't have just made it up in order to provide an amazingly good answer, so how can it give a false statement of fact and claim that it didn't get the information from anywhere? (I.e. what mechanism does it use to generate facts if they don't originate somewhere else?)

And worse, it actually lied while apologizing:

I will make sure to double-check my sources more carefully in the future.

  • 2
    $\begingroup$ You should not trust anything that ChatGPT says, its a Stochastic Parrot, and you are asking us about mechanisms, when there is no public information about its inner workings. $\endgroup$
    – Dr. Snoopy
    Commented Feb 24, 2023 at 9:15
  • 1
    $\begingroup$ The current term for this phenomenon is "hallucination" . It is very easy to induce, once you know how. medium.com/@cooper.white_86633/… $\endgroup$ Commented Feb 24, 2023 at 20:51
  • $\begingroup$ From what I experimented Chatgpt is not very good with movie. It seems to check only a few term of the question and return something that is induced from these terms, not the full question. $\endgroup$
    – Ptit Xav
    Commented Mar 5, 2023 at 16:10
  • $\begingroup$ I think the term parrot is very misleading. It is not small, it is not in a cage and it moves as a lot of us move. I prefer the name myrianthrope. $\endgroup$ Commented Apr 16, 2023 at 7:47

2 Answers 2


The reason is that ChatGPT is not a data retrieval system. In other words, it is not consulting a database, nor is it performing searches of the internet, to find answers to questions posed.

Instead, it is generating novel answers based on the trained parameters within the network. In a sense, this is far more impressive.

If you think about it, ChatGPT and large language models generally provide a very interesting method of compressing data. Of course, compression isn't the point of LLMs, but you can view this as one of the outcomes. How so?

During training, the parameters are learned such that the model is able to reasonably predict the next most likely token (word/bigram/trigram/letter/symbol) based on all of the tokens that have come before it in that session, including any base prompt that is included silently. When you ask it to recite something like The Lovesong of J. Alfred Prufrock, it can do so very accurately... yet, it does not have a copy of Prufrock "memorized" somewhere. Instead, it is recreating Prufrock generatively.

This is also why its answers, while incredibly confident, can be wildly incorrect. It is essentially generating the most likely token that comes next, not reasoning or thinking about what the text means.

  • $\begingroup$ How can we assess to what extent it compresses or rather memorizes data? $\endgroup$ Commented Feb 24, 2023 at 11:29
  • 1
    $\begingroup$ That's a super question. Intuitively, we could evaluate the billions of parameters in the model and attempt to recreate data from them directly... However, since they are in an encoded representation (I would think of it as an eigenvector of some high dimensional space), that would be super challenging. Definitely an interesting thing to study though! $\endgroup$ Commented Feb 24, 2023 at 11:45
  • 1
    $\begingroup$ its also funny that it claims it's going to "double-check" its sources in the future, considering it has no sources and it's not performing any sort of review on what it says $\endgroup$
    – Ivan Viti
    Commented Mar 22, 2023 at 11:11

Incalculable times i caught it literally making things up it does not know. For example an hour ago i asked it if Arnold Schwarzenegger says the line "i don't want to hurt you" in any of his movies. It first correctly returned no, then i wrote try harder and it apologized and said he says it in Batman and Robin 1997. I asked for exact minute it said 1:03, i looked no such line, then it said it's at 1:26 then i looked and there said there is no line there either. Then it apologize again and said there is no such line. So It lied 3 times, totally making it up. And i have personally caught it lying already countless times.

  • $\begingroup$ Even at technical questions, it really wants to please. Coprime Matroid looks like a formal mathematical proof, but some of the smaller details don't actually make sense. It would likely fool a non-mathematician. What's really impressive though is that even though there is actually no such thing, it invented the name "the coprime matroid", which would have been the perfect name for it had such a thing actually existed, making it look like it really had researched the answer and found various references to it using this name. $\endgroup$ Commented Apr 16, 2023 at 0:18
  • $\begingroup$ I know, i been talking to it about physics and electrical engineering, when it does not know it simply makes it up and pretends it's a fact altho it is 100% bs. Like this guy puts it "it creates authorative bullshit" youtu.be/1wzPr4cUoMQ?t=479 $\endgroup$
    – Nix
    Commented Apr 17, 2023 at 5:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .