15
$\begingroup$

Having used various AI bots often over recent months, I noticed that often it will claim to know something, even if it doesn't. It would then either explain something which is clearly nonsense, or by rambling on about how the answer isn't known in general. Or how, if asked for example- "would you be able to explain X" it wouldn't respond "yes, I could" but rather would elucidate X. Have they been trained to always respond as though it were a know-it-all? (Google's Bard and ChatGPT specifically, although I'm assuming only open-source AI will be answerable)

$\endgroup$
7
  • 8
    $\begingroup$ We would do well to train users that "AI" stands for "Artificial Idiot". Idiot savant if they are lucky. $\endgroup$
    – nigel222
    Aug 30, 2023 at 13:59
  • 15
    $\begingroup$ What nobody wants to admit is that the language used by a large language model naturally mimics the language style and tone of the humans whose utterances were used to build the model. I personally find that ChatGPT (for example) behaves exactly the same as I have observed some humans behave. $\endgroup$
    – user21820
    Aug 30, 2023 at 14:35
  • 6
    $\begingroup$ Humans work the same way. Some humans raise above the mould and can do more self-reflection and cross-analysis of the data they're fed, but that's still a bit out of reach of contemporary AI. Inattentive humans absorb all the data they're fed uncritically and only rarely realize and correct after the fact. Many unscrupulous people exploit that "feature" of human brains - advertising, misinformation and all sorts of other manipulation. Thinking you understand everything (especially something you know little about) is unfortunately very natural to our brains :) $\endgroup$
    – Luaan
    Aug 31, 2023 at 6:25
  • 4
    $\begingroup$ Would you really want a Q&A service to answer "Could you explain X?" with "Yes, I could" instead of an explanation of X? How is that any less snarky? $\endgroup$
    – user253751
    Aug 31, 2023 at 14:12
  • 2
    $\begingroup$ “often it will claim to know something, even if it doesn't” — are you implying that there are other things an AI does ‘know’? $\endgroup$
    – gidds
    Aug 31, 2023 at 17:51

6 Answers 6

25
$\begingroup$

Have they been trained to always respond as though it were a know-it-all?

Yes, sort of, although it is not in some attempt to upset you or other users.

The language models used in the chat bots have been fine-tuned in conversations with users, and also have been "pre-prompted" to be helpful assistants. This impacts the kind of text that they generate.

Separately to this, the fact that they are often incorrect or generate nonsense is a limitation of the core technology. There is no underlying world model, the model cannot self-assess whether anything it generates is correct. Although the large language models can often surprise users (including the original developers) in the breadth of what they can respond to correctly and give the appearance of understanding, the same models can also fail easily at apparently simple tasks. This can cause some cognitive dissonance for users who expect an AI that appears to communicate like an intelligent human to be able to perform simple maths for example.

There is probably no global default chat tone that would suit all users. You could however use the raw model more directly and create a chatbot with a different tone to suit you. This would not impact accuracy, and would not add ability to self-assess correctness, but it may address feeling of being talked down to for example. Some services, like Character AI, attempt to give you tools to do just that, although the level of usefulness you get from them will depend on what they focus on (Character AI is more for creative fun, than for functional assistants).

In limited cases you can also address accuracy with pre-prompts or fine tuning that put a few facts into working memory. This is limited, and cannot address lack of general ability in logic or maths though. Corporate users can do this to give a chat bot some basic product knowledge or correct address for their help desk etc.

$\endgroup$
5
  • 5
    $\begingroup$ I think adjusted the tone of an ai chatbot won't really change anything because the bot simply doesn't know whether it is knowledgable on a topic or not. OP wants a promp something like 'explain this if you are able to do it but tell me if you are not'. The bot simply can't make this distinction. $\endgroup$
    – quarague
    Aug 30, 2023 at 6:28
  • 2
    $\begingroup$ @quarague It won't change factual accuracy, because the bot cannot realistically introspect about what it is outputting. However, it could remove the "know it all" vibe that they are complaining about. $\endgroup$ Aug 30, 2023 at 6:48
  • 2
    $\begingroup$ If it were to start everything with 'I'm not really sure but I believe ..' this doesn't really help OP because it would do that everywhere including on topics where it is 100% perfect spot on. $\endgroup$
    – quarague
    Aug 30, 2023 at 6:50
  • 1
    $\begingroup$ @quarague I am not seeing that the OP has asked for that. I am reading that they are literally complaining about tone not matching the capability. I also cover this difference in the third paragraph, starding with "separately to this", and I make it clear that the model cannot self assess. I am not sure what edit/change to this answer you are suggesting therefore? $\endgroup$ Aug 30, 2023 at 6:58
  • $\begingroup$ Comments have been moved to chat; please do not continue the discussion here. Before posting a comment below this one, please review the purposes of comments. Comments that do not request clarification or suggest improvements usually belong as an answer, on Artificial Intelligence Meta, or in Artificial Intelligence Chat. Comments continuing discussion may be removed. $\endgroup$
    – Dennis Soemers
    Aug 31, 2023 at 14:50
19
$\begingroup$

Humans, who know things, tend to limit themselves to speaking of things they know, especially in print. Thus any text corpus you may find will have an overpopulation of examples of confidently presented answers.

Chatbots, which do not know things, cannot impose a similar limit and thus will babble at length on topics about which they know nothing. But because they are trained on those data of people writing about things they know, they will tend to mimic the same confident tone and style of those people.

It is very important to reiterate here again that chatbots do not know things. By good fortune, they may produce a text which happens to convey factually accurate information. But they may not, and because they do not know things, they do not in advance (or even in retrospect) if they can (or did) produce a factually accurate text or meaningless drivel.

Thus a chatbot (or its developer by proxy) has only two options available when asked a question. Option 1 is to attempt to generate an answer to every question. Option 2 is to answer every question with "I don't know." The latter is technically uninteresting and practically useless, so everybody chooses Option 1.

$\endgroup$
2
  • 7
    $\begingroup$ This is mostly incorrect. Mechanistic interpretability research has proven that transformers store factual information (which can be edited) in their MLP blocks and hence, GPT-based chatbots do actually know things (for references, google "Knowledge Neurons in Pretrained Transformers" and "Moving the Eiffel Tower to ROME"). They also "know" when they generate falsehoods and can be steered to the truth by activation engineering (google "Discovering Latent Knowledge Without Supervision", "The Internal State of an LLM Knows When its Lying" and "Eliciting Truthful Answers from a Language Model"). $\endgroup$
    – ain92
    Aug 31, 2023 at 1:06
  • $\begingroup$ Also, Option 1 vs Option 2 is a false dichotomy. The chatbot could say, "I am not sure but..." and proceed. $\endgroup$ Sep 1, 2023 at 22:05
14
$\begingroup$

It's quite simple really. LLM based generative AIs don't "know" anything. They're glorified next-word predictors. The only way that they're going to produce an answer to a prompt along the lines of "I don't know" is if their training corpus would indicate that that was the likely response a human would give to the same prompt. Admitting that we don't know something isn't a typically common human trait in the first place, and publishing that online is even less so.

$\endgroup$
1
  • 1
    $\begingroup$ And they don't "feel the need" to act in any way either. They're just optionally (cons)trained in a particular direction. $\endgroup$
    – sehe
    Aug 31, 2023 at 18:46
7
$\begingroup$

The only thing the bot "knows" is how to generate sequences of lexical symbols (ie. texts) that mimic what people have written in the corpus it has been trained on.

The output is based on the prompt using complex rules + some internal state (so that it seems to "remember" the past discussion). It is all just mathematics that is used to choose which symbols (ie. letters and punctuation) to output. It could be implemented just as well with pocket calculators, or even doing the math manually, albeit very slowly and very arduously.

There is nothing else to the bot. No knowledge, no reasoning, no needs, no goals, no personality. Basically just numbers manipulated in a complex way.

Whatever else you happen see in the output is always your interpretation. If you spot a need in the text, it is most likely there only because some actual person did write something that reflected their need, and from it, the bot learned such rules that the output mimics something that was written with emotion.

It is as sentient as a cartoon character in an animated movie. All emotions that you assume to be in the character/text originates from you.

So, why it looks like they feel the need to be know-it-alls? Because they mimic what people write, and people are like that.

$\endgroup$
2
  • $\begingroup$ Just because the model outputs know-it-all text for a set of queries does not necessarily imply that the training dataset is equally know-it-all-esque. (c.f. Most famously, see the "debate" on Yann LeCun's tweet.) $\endgroup$ Sep 1, 2023 at 6:07
  • $\begingroup$ If we think of humans as very complex models/functions, we may observe that trained humans are prone to certain behaviors (e.g. seeking food/water/shelter, tribalism, groupthink, ...) by nature (i.e. model DNA) in addition to nurture (i.e. training data). $\endgroup$ Sep 1, 2023 at 6:17
1
$\begingroup$

The main reason an AI model won't tell you "I don't know" is that the developer doesn't want you to get the impression that the model is incapable of answering your questions.

Imagine a scenario where you had several difficult or non-trivial questions that the model doesn't have the right answers for. If answers in the form of "I don't know" kept piling up, it will drive you away and make you less confident about getting an answer the next time you have a question.

So even if it gives an incorrect or inaccurate answer, by just giving an answer, it leaves you under the impression that sometimes it gives correct answers - sometimes it doesn't, and the next time you have a question you might return because you'd think it might give you a correct answer this time.

$\endgroup$
-2
$\begingroup$

The large language models underpinning these "AI" bots have indeed been tweaked to be biased against providing a negative response.

The simple reason being that the executives of the companies trying to sell "AI" as the Next Big Thing, believe they're less likely to make sales if their product appears to be unable to answer questions. Thus they instruct their engineers to train these models to divert, dodge and dissemble - even to plainly ridiculous lengths - when the model is incapable of coming up with an affirmative answer to the question asked.

$\endgroup$
1
  • $\begingroup$ I f you can provide a source for that, I'm sure you'll get upvoted. Seems it sounds a bit too conspiracy like for the community... $\endgroup$ Sep 2, 2023 at 18:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .