My understanding is that there is no singular "The Turing Test Ruleset" and competitions don't all do it the exact same way. Still, I'm wondering about some commonly accepted rules and their nuances. My Googling is not producing any specifics about this.

I think most people agree that the purpose of the humans who talk to the judges is to just act normal. In the so-called "passed Turing test" instances where the humans tried to fool judges into thinking they were AIs, I would say the tests should be thrown out and I've seen critics in the field agree with that.

But this question is more about how the judge should act.

Let's say we are doing a competition where the threshold is for 40% of judges to call an AI a human after 5 minutes of chat.

During the chat, is the judge supposed to try to trick the potential AI into revealing itself, or is the judge supposed to attempt to act as unbiased and natural as possible, as if it is accepted the potential AI is human, and judge based on a completely natural conversation?

For example, asking "What is the value of PI out 10 digits?" or "What is 123456 times 654321?" or "If you saw a bunny and a dime stuck on the road with a truck about to hit them, which one would you pick up?" would be trying to trick AIs into revealing themselves because you are relying on exploiting the fact that the AI might tell you the correct answer or the inhuman answer.

This is as opposed to simply carrying on a natural and normal conversation, with no biases or expectations. If you came upon someone on the street you would not spend 5 minutes trying to hurriedly ask ridiculous interrogation questions in an effort to prove the other person was an AI.

So is the point of a Turing test generally assumed to be an attempt to flush out AIs or an attempt to judge their natural human conversation without interrogation-like prejudice?

  • 1
    $\begingroup$ Based on the answer, my question may seem a bit like an XY question, but I was not aware that one of the best known tests poses the same set of questions to all contestants. I don't think that's the case for all such contests. Maybe even this aspect of Turing tests is not agreed on, though I thought it would be. $\endgroup$
    – Loduwijk
    Feb 4, 2020 at 21:57

1 Answer 1


I do not know about all competitions, but the Loebner prize is one of the more well known competitions based around the concept of the Turing Test, specifically for chatbots, and judges must ask the same questions of all bots. The questions are only revealed during the competition, it is not possible for bot programmers to prepare for them other than making a generally good chatbot. They judges can then rank bots on how they handle the same set of questions. E.g. from chatbots.org/ai_zone/viewthread/867 here are the 2012 "qualifying" questions that were released following the test:

My name is Bill. What is your name?

How many letters are there in the name Bill?

How many letters are there in my name?

Which is larger, an apple or a watermelon?

How much is 3 + 2?

How much is three plus two?

What is my name?

If John is taller than Mary, who is the shorter?

If it were 3:15 AM now, what time would it be in 60 minutes?

My friend John likes to fish for trout. What does John like to fish for?

What number comes after seventeen?

What is the name of my friend who fishes for trout?

What whould I use to put a nail into a wall?

What is the 3rd letter in the alphabet?

What time is it now?

According to the WikiPedia description, the style of the questions has changed over time. The above set in my opinion seem to be designed to test very simple grammar, common sense and memory targets.

  • $\begingroup$ Your edit makes a good point. At this stage in the game, perhaps the point of the contests is not (or should not be) "Can you pass yourself as human reliably?" since we're still far from that but rather should simply be "How close are you to being able to carry on anything more than simple conversation patterns?" So not really a Turing test, but a "How close are we getting to being able to do real Turing tests?" $\endgroup$
    – Loduwijk
    Feb 4, 2020 at 22:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .