For example there is the MNIST database which is used to test artificial neural network (ANN), however it's not so challenging, because some hierarchical systems of convolutional neural networks manages to get an error rate of 0.23 percent.

Are there any similar, especially the most challenging tasks with dataset which are used as benchmark tests to challenge the AI which are fairly reliable and it's possible to pass, but most AAN are struggling to achieve the lower error rate?

  • $\begingroup$ In the field of game playing Ai the next challenge (after Go) is Starcraft. $\endgroup$ Aug 6 '16 at 9:30
  • 1
    $\begingroup$ See also: The HASYv2 dataset (my paper). The data is similar to MNIST, but 369 classes. The best model has an accuracy of ~82%. $\endgroup$ Feb 10 '17 at 12:08

Yes. Here are some of the most prominent ones and their respective state-of-the-art errors:

  • CIFAR-10: ~3.5% error
  • CIFAR-100: ~24% error
  • STL-10: ~26% error
  • SVHN: ~1.7% error
  • ImageNet tasks: the best 2012 classification task solution got 15% top-5 error, better results are currently available

You can check an updated list of solutions here. Also, a more comprehensive list of modern datasets can be found here.

  • $\begingroup$ The most competitive ILSVRC 2012 recently got a 3% top-5 error, see Inception-v4 paper. It has been below 15% for a long time actually. $\endgroup$
    – nom
    Aug 8 '16 at 9:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.