How do language models know what they don't know - and report it?

Question

Again and again I ask myself what goes on in a pre-trained transformer-based language model (like ChatGPT9) when it comes to "know" that it cannot give an appropriate answer and either

• states it ("I have not enough information to answer this question.")

• asks for more specific information ("Please tell me which kind of XY you mean.")

• calls a plugin (like Wolfram or ScholarAI)

(I assume that this will never happen without reinforcement learning by human feedback. A pre-trained-only model would always answer something (possibly hallucinating) and not "reflect" about its lack of knowledge.)

The only possibility that I can see - but it's not really explanatory: that after some steps of execution the sum of the top_k probabilities of the final vector (which gives probabilities to the all words in the vocabulary) is too small. But what, when this happens only late? ChatGPT would already have produced lots of words - but one never observes that he stops generation after some lengthy text and only then ends with something like "Ah, finally I see that I'm missing information. I wasn't aware in the beginning." ChatGPT immediately admits that he doesn't know (when he does). And when ChatGPT calls a plugin - e.g. ScholarAI - he does it without having produced a single word of response to the last message.

In principle, ChatGPT could generate a complete response in the background that then is checked somehow if it's "satisfactory". If yes it's given as output (simulating word-by-word generation), if not, it's regenerated with some sort of trigger (a hidden token?) to admit that ChatGPT is missing information or to call a plugin.

What's the clever trick under the hood (in some technical detail)?

Answer 1

The data it is trained on includes variants of "I don't know". For instance, if you ask me what is the meaning of life and I reply I don't know, then that is the information schema the AI absorbs. It knows what it does not know, in the same way that it knows what it knows.

Here is another way to look at it. When in training, people were asked to interact with GPT 3.5. At that time, the trainers would have received many incorrect responses. They would then flag to GPT that the response is incorrect/inaccurate from which it would learn to either flag the issue to the user upfront (I am only a LLM ... etc. etc.) or to say I do not know, or some variant. In all of these, the chat interactions and the training data enable it to learn the association of a sequence of words and the idea in it with the phrase "I don't know" or "I won't answer that" in the same way that it learns anything else. The guide rails are programmed in through the interactions with human trainers. This way the math does not change. You only need human trainers to interact with the system and they implicitly program the guide rails.

The schema is plugin aware but not trained on any specific plugin. The following is the information flow (from https://platform.openai.com/docs/plugins/introduction) taken verbatim from the webpage:

• OpenAI will inject a compact description of your plugin in a message to ChatGPT, invisible to end users. This will include the plugin description, endpoints, and examples.

• When a user asks a relevant question, the model may choose to invoke an API call from your plugin if it seems relevant; for POST requests, we require that developers build a user confirmation flow to avoid destruction actions.

• The model will incorporate the API call results into its response to the user.

• The model might include links returned from the API calls in its response. These will be displayed as rich previews (using the OpenGraph protocol, where we pull the site_name, title, description, image, and url fields).

• The model can also format data from your API in markdown and the ChatGPT UI will render the markdown automatically.

In a nutshell, ChatGPT is intelligent but its plugin system is very limited.

Answer 2

It makes sense to assume that reinforcement learning from human feedback (RLHF) has some merit, at least. I'll explain myself.

In RL we have a reward (the human feedback), a policy (which should be GPT itself), and (one or two) value functions. Here it say that in RLHF it learns reward model (RM), i.e, a network that takes a (generated) sequence and the human feedback, and outputs a numerical reward that represents the human preference (I think how much the preference is followed.) The RM scores/evaluates the goodness of the generated text, basically.

Since the RM specifies a reward function, once it's learned you can use standard RL to optimize the language model to follow the RM and thus the human preferences.

Then at inference time, one can still use the RM to evaluate the generated text and so if not satisfactory (i.e. predicted preference is too low), they can fallback to another (specialized) LM or even have some special instructions that ask the user for clarification. This is my guess.