2
$\begingroup$

I ran into this AI-SE question from 5 years ago and I believe that an updated version could be interesting to discuss nowadays: Is the smartest robot more clever than the stupidest human?

Today's best LLMs are displaying a lot of human-like abilities: proficiency in natural languages, ability to code, logical reasoning, role playing and so on. They can even solve CAPTCHAs, design games, answer questions about stories or write new ones: these were the "shortcomings of robots" in 2018, according to the answers to the question I linked.

Question

How do the best LLMs of today compare to a "dumb human"? In what tasks are all normal humans still better than AIs? Is there any test that every able-bodied human would pass, but top LLMs would still fail?

Definitions and clarifications

A "dumb human" is a person without recognized disabilities or obvious problems, who doesn't have particular skills and who is considered not very intelligent (low IQ).

Of course the LLMs available to the public have a number of objective limitations: they can only process text-to-text, they work with tokens rather than characters, context length is just few kilo-tokens, and they have no long-term memory. However a number of open source projects have shown various solutions to these problems, and the non-public version of the commercial LLMs already support much larger context windows, image input and similar features. Observations like "LLMs can't move arms as they don't have it", "LLMs fail to count characters because they're token based", "LLMs can't speak nor listen to speech" are not interesting.

Improve this question
$\endgroup$
2
  • $\begingroup$ Looks like a duplicate of ai.stackexchange.com/questions/7021/…. If it isn't please explain why not. $\endgroup$
    – Bruce Adams
    Commented Jun 15, 2023 at 18:40
  • $\begingroup$ @BruceAdams: that question is about general capacities of robots in 2018 and the answers say those robots aren't able to solve CAPTCHAs and write stories. This one is about text-based tasks of LLMs in 2023. $\endgroup$
    – Blue Nebula
    Commented Jun 15, 2023 at 19:49

2 Answers 2

Reset to default
6
$\begingroup$

LLMs seem to be limited at "compositional tasks." Have a look at this paper, in which the authors

investigate the limits of these models across three representative compositional tasks—multi-digit multiplication, logic grid puzzles, and a classic dynamic programming problem. These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer.

I don't know if a "dumb human" can do dynamic programming problems, but

humans can solve 3-digit by 3-digit multiplication arithmetic after learning basic calculation rules. Yet, off-the-shelf ChatGPT and GPT4 achieve only 55% and 59% accuracies on this task, respectively.

I gave ChatGPT two tries (what is "311 times 877") and ("what is 513 times 799"), and it got them both wrong.

Another example from that paper is a "zebra" or "einstein" puzzle. Look at page 18 in the paper for an example. (I'll copy it below) It took me a couple minutes to solve. ChatGPT tries its best, but gets it quite wrong.

the puzzle

There are 3 houses (numbered 1 on the left, 3 on the right). Each has a different person in them. They have different characteristics:

  • Each person has a unique name: peter, eric, arnold
  • People have different favorite sports: soccer, tennis, basketball
  • People own different car models: tesla, ford, camry

clues:

  • The person who owns a Ford is the person who loves tennis.
  • Arnold is in the third house.
  • The person who owns a Camry is directly left of the person who owns a Ford.
  • Eric is the person who owns a Camry.
  • The person who loves basketball is Eric.
  • The person who loves tennis and the person who loves soccer are next to each other.

Match the person, favorite sports, and car models to each house.

ChatGPTs output (for me just now)

GPT's full output

<omitting the output where it "thinks">

Therefore, the matching of people, favorite sports, and car models to each house is as follows:

House 1:

    Person: Peter
    Favorite sport: Tennis
    Car model: Tesla

House 2:

    Person: Eric
    Favorite sport: Basketball
    Car model: Ford

House 3:

    Person: Arnold
    Favorite sport: Soccer
    Car model: Camry

and its easy to see that some of the constraints are violated. (e.g. the first one: Ford goes with Tennis)

Improve this answer
$\endgroup$
7
  • $\begingroup$ Interesting to note that you've only found weaknesses in LMMs where traditional AI would have no problem. $\endgroup$
    – Bruce Adams
    Commented Jun 18, 2023 at 9:12
  • $\begingroup$ Here is a paper on one of several ongoing attempts to marry the two arxiv.org/abs/2305.12295 $\endgroup$
    – Bruce Adams
    Commented Jun 18, 2023 at 9:14
  • $\begingroup$ @bogovicj: GPT-4 was able to compute "311 times 877" on the first attempt: chat.openai.com/share/bca563aa-09ae-44e4-aea2-1463f7d3ce77 . It failed to compute "513 times 799", but it succeeds on other attempts or if you ask it to break the computation in even smaller steps. Similarly, it can solve the puzzle, if you ask it to come up with a formal way to model the problem, then solve it step by step. Would a "dumb human" be able to solve these problems more reliably than GPT-4? I tend to believe that today's top LLMs are better than low IQ individual at these tasks. $\endgroup$
    – Blue Nebula
    Commented Jun 18, 2023 at 16:03
  • $\begingroup$ Actually, I looked again at the computation of 311*877. There's a mistake in there ("273200 is 311*8", when it should be "248800"). In spite of that the result is surprisingly correct (how?!). Nevertheless, it should be able to solve these problems if you let it "reason" on it enough, through its output. $\endgroup$
    – Blue Nebula
    Commented Jun 18, 2023 at 16:10
  • $\begingroup$ @BruceAdams that looks like an interesting paper, thanks for sharing. Just want to clarify some small things: 1) I didn't those weaknesses, its the papers' authors that study them. 2) Those are the "only" ones in that paper, which is not an exhaustive list, of course. 3) One doesn't need "AI" to do multiplication or satisfy constraints; regular old comp sci can do it. $\endgroup$
    – bogovicj
    Commented Jun 19, 2023 at 19:23
0
$\begingroup$

Humans (any human) are still better at one of the most important task (whether it is a text-based task or not).... Humans can give you their own opinion or "sentiment" about a text. This means any human can tell you if he/she likes the text or not. An LLM don´t have any kind of own "personality", "opinion" or "feelings" so it can not give you its own opinion but a "general" sentiment (which is generally mentioned as Sentiment Analysis in NLP tasks) based on all the training data used by the bot.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ This is tricky. You can definitely tell an LLM to roleplay as a certain personality or character, and it's opinion as observed from the output would meet criteria of having a specific self-consistent view. You can then get into the weeds about whether such output is a real measure of something internal, or more like text puppetry. IMO this gets into "it has no arms" territory - so what if the raw model has no personality, if it can realistically assume a personality on demand? At least when we're talking about input/output capability $\endgroup$
    – Neil Slater
    Commented Jun 18, 2023 at 15:40
  • 1
    $\begingroup$ @RaulAlvarez: can you come up with a test to verify what you said? LLMs are able to share a sentiment or opinionated impressions about a text. To get commercial LLMs that have been through RLHF to do this, you need to ask them to role play. But LLMs pre-RLHF do it spontaneously. $\endgroup$
    – Blue Nebula
    Commented Jun 18, 2023 at 15:54
  • $\begingroup$ Maybe you can ask any LLM if they like a chocolate ice cream or a strawberry ice cream. Same about a text. You can ask him if he prefers Real MAdrid to Barcelona, red or blue color.... he will not give you his opinion... nor his preference. $\endgroup$
    – Raul Alvarez
    Commented Jun 18, 2023 at 17:58
  • 1
    $\begingroup$ @RaulAlvarez The LLM will give an answer though, and it will typically be a popular one, plus if challenged on why it made the choice, it will be able to explain. When it does this, you have to perform extra tests, or make assumptions about the internals of the model in order to back up your claim that the model doesn't really have a preference or an opinion. IMO this comes under the OP's request not to reference things that the model does not have . I.e. not having a true opinion is the same as not having arms to wave $\endgroup$
    – Neil Slater
    Commented Jun 19, 2023 at 7:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .