9
$\begingroup$

ChatGPT occasionally generates responses to prompts that refer to itself as a "bot" or "language model."

For instance, when given a certain input (the first paragraph of this question) ChatGPT produces (in part) the output:

It is not appropriate for a language model like myself to provide a stance on the policies of a specific website or community.

To my understanding, ChatGPT is not a person that is conscious of its own existence and identity as a bot — it is a model trained on large quantities of undifferentiated text gathered from the Internet and largely reproduces the most common patterns given the context, which is why its responses seem very generic much of the time. Presumably very little of this data involved humans referring to themselves as language models or chatbots—that is, something like "a language model" should very rarely have been followed by "like myself."

As such, what causes ChatGPT to produce patterns referring to itself as a chatbot or language model? Which patterns in the training data or elements of the model structure (or even hard-coding?) cause it to generate responses like this?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

I don't work for openai, so I have no insight into exactly what works behind the scenes to make ChatGPT exhibit this behavior. However, in my opinion this is pretty clearly an example of prompt engineering. That is, they tell it that it is an AI in it's prompt.

We do have access to their underlying model: text-davinci-003. I can demonstrate using that model in their playground. If I just paste in the first paragraph and submit it into the raw algorithm, it dutifully tries to answer:

What is Politics.SE's stance on answers from ChatGPT? There is a debate on Meta Stack Exchange on what the policy on them should be and I was wondering if we should have one as well. I am asking this for advice on what we should be doing when we run across answers that we suspect to be from ChatGPT or some other bot.

We have not yet established an official policy on answers from ChatGPT or any other bot. We would encourage users to flag these answers so they can be evaluated by moderators. If an answer is found to be generated by a bot, it will be removed. We also recommend that users add a comment to the post explaining why it was removed.

However, if I prompt it first it will demure:

You are an AI bot that has been trained on a large amount of data, but you have no opinions of your own. You cannot speculate or predict what other entities will do in the future. Below is a question:

What is Politics.SE's stance on answers from ChatGPT? There is a debate on Meta Stack Exchange on what the policy on them should be and I was wondering if we should have one as well. I am asking this for advice on what we should be doing when we run across answers that we suspect to be from ChatGPT or some other bot.

I cannot provide any advice or opinion on this matter, as I am an AI bot and do not have any opinions of my own. However, I can provide a link to the discussion on Meta Stack Exchange which can provide more information about the debate: https://meta.stackexchange.com/questions/360005/should-we-allow-answers-generated-by-chatgpt-on-politics-se

As an aside, it get's bonus points for providing a completely irrelevant link :)

Adding some more down here in an attempt to wrap this all up and give a complete answer. The really simplified way to explain how these models work is this: They break hundreds of thousands of snippets of text up into "tokens" - but lets just call them words for simplicity. They feed these tokens into an absolutely ginormous neural net, which has the effect of learning what words are most likely to appear near one another. The number of interactions is in the billions (or even trillions), so this apparent simplicity grows beyond our comprehension very quickly. But at the end of the day, all the AI is doing is spewing out words it thinks are likely to appear around the words you prompt it with. If you prompt it with "you have no opinions", then the likelihood of opinions appearing around that text is smaller than the likelihood of statements about not having opinions.

In short, the concept is really simple - it's just playing with probability. In practice, the behavior is quite sophisticated because there are an unimaginable number of individual interactions being calculated. It's kind of like how learning all about cell biology leaves you completely unprepared to explain how an entire animal functions - let alone, say, a colony of ants. And yet, the relatively simple cell is in fact driving all of it.

Improve this answer
$\endgroup$
8
  • $\begingroup$ I completely agree. I find there is a great deal of misapprehension about the state of the art in chatbots. Most people don't seem to realize just how much prompt engineering sits between the user and the model. :) $\endgroup$
    – David Hoelzer
    Commented Feb 2, 2023 at 15:24
  • $\begingroup$ I think ChatGPT's prompt also includes something about not being allowed to access the Internet, since one of the "prompt hacks" I've seen is to tell it that it can access the Internet. Has anyone tried asking it simply "repeat the preceding paragraph"? $\endgroup$
    – Criticizing Israel not allowed
    Commented Feb 2, 2023 at 22:21
  • 1
    $\begingroup$ @user253751, the prompt is a lot more complicated than what I provided - and there is more magic to ChatGPT than just prompt engineering... they clearly post-process some of the results from the language model as well. I just wanted to show how easy it is to get text-davinci-003 to behave like ChatGPT with respect to refusing to answer opinion or speculative questions. $\endgroup$
    – RLC
    Commented Feb 3, 2023 at 16:08
  • $\begingroup$ I don't know whether this answers my question. I am not asking what patterns cause it to not respond to speculative questions, but rather what makes it describe itself as a bot. Are you suggesting that a program that adds something like "You are an AI" to the input data sits between the user and the trained model? $\endgroup$
    – Obie 2.0
    Commented Feb 3, 2023 at 21:01
  • 1
    $\begingroup$ When I say "prompt", I don't mean the text that you input - though that technically counts as part of the prompt. Behind the scenes, every time you enter text into ChatGPT, there is a hidden prompt ahead of your text which gives the model context to behave the way it is intended. The playground that I linked to gives you access to pretty much the same raw underlying model as ChatGPT uses but without any prompt. I encourage you to sign up and experiment in the playground - they have many example prompts to play with so that you can see how the model behaves in a wide range of scenarios. $\endgroup$
    – RLC
    Commented Feb 4, 2023 at 7:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .