One way of ranking human intelligence is based on IQ tests. But how can we compare the intelligence of AI systems?

For example, is there a test that tells me that a spam filter system is more intelligent than a self-driving car, or can I say that a chess program is more intelligent than AlphaGo?


There are different ways to compare different kinds of AI techniques.

As a starting point, be aware that "AI System" can mean an incredibly broad range of things. In popular culture, we usually think of a deployed system that uses AI techniques. These systems can only be compared on the basis of their performance, and their performance may have relatively little to do with AI itself (e.g. their behaviours might be more strongly affected by user interface decisions, not the AI techniques under the hood).

In contrast, AI researchers are usually more interested in comparing the performance of different AI algorithms at solving the "AI-ish" parts of the problem a fully developed system aims to solve. A common way to do this is with benchmark problems. For example, in machine learning it is common to compare two algorithms by running each of them on a commonly used dataset, and comparing the performance of the models they create. In AI Planning, it is common to issue planning challenges to the community, and compare the quality of the plans on several different axes (e.g. average wait times, maximum wait times, whether goals were accomplished, how long it took to create a plan, etc.).

There is no generally agreed upon way to compare techniques across different areas of AI, but a commonly adopted approach is the Turing Test. In the Turing Test, we care only about the ability of the system to mimic something like human intelligence. It's fair game to ask about planning problems, or learning problems, or other topics, so you could in some sense judge one technique to be better than another. However, most judgements made in the Turing Test are subjective, so it's not clear that it really solves the problem you posed.

  • $\begingroup$ But as per the Turing Test. The machine which behaves like a human that machine is an intelligent one. This is what my understanding about the Turing Test $\endgroup$ – Maheshwar Ligade Sep 7 '18 at 5:51
  • $\begingroup$ +1 for the mention of performance on the job itself, rather than a general intelligence measure. $\endgroup$ – DuttaA Sep 7 '18 at 7:03
  • $\begingroup$ @MaheshwarLigade That's a popular definition, but I think most AI'ers would disagree at a least a little. The problem is: what does "behave like a human" mean? Who decides that? It turns out that very simple programs like Eliza can meet some people's bar. On the other hand, very complex modern attempts at the Turing Test are still pretty easy for researchers to detect. Some of the more successful ones get by pretending to be 5 year olds (i.e. less intelligent!). Check out the Loebner Prize results for more. $\endgroup$ – John Doucette Sep 7 '18 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.