1
$\begingroup$

ImageNet dataset is an established benchmark for the measurement of the performance of CV models.

ImageNet involves 1000 categories and the goal of the classification model is to output the correct label given the image.

Researchers compete with each other to improve the current SOTA on this dataset, and the current state of the art is 90.88% top-1 accuracy.

If the images involved only a single object and background - this problem would be well-posed (at least from our perceptual point of view). However, many images in the dataset involve multiple objects - a group of people, a person, and an animal - the task of classification becomes ambiguous.

Here are some examples.

The true class for this image is bicycle. However, there is a group of people. The model that recognizes these people would be right from the human point of view, but the label would be wrong.

enter image description here

Another example is the fisherman with fish called tench. The model could have recognized the person, but be wrong.

enter image description here

So, my question is - how does much the performance of the best models of ImageNet reflect their ability to capture complicated and diverse image distribution, and how much the final result on the validation set is accidental. When there are multiple objects present on the image, the network can predict any of them. Prediction can match the ground truth class or can differ. And for the model, that happens to be luckier, this benchmark will show better performance. The actual quality can be the same, in fact.

Improve this question
$\endgroup$
10
  • $\begingroup$ Are these images really from ImageNet? I can't believe that images like these are in a dataset for image classification. Maybe you can provide a reference to the images, maybe a piece of code that loads ImageNet and displays them. I understand your concerns but the question "how much does the prediction of the model conform with one of the multiple labels" is a bit unclear, although I understand where you want to go with this question. Maybe you can reformulate it in a clearer way. $\endgroup$
    – nbro
    Dec 23, 2021 at 23:51
  • $\begingroup$ @nbro These are from ImageNet. The upper image was randomly sampled from train dataset. Image with fish is the first image in train dataset. Note, that ImageNet is available only by request, one cannot simply download it (not just due to the large size). $\endgroup$
    – spiridon_the_sun_rotator
    Dec 24, 2021 at 4:49
  • $\begingroup$ Ok. I understand the problem(s) that you bring up: we're trying to force the model to say to which class the object in the image belongs, while, in many images, there is clearly more than one object. However, I still think this question is not fully clear. Are you trying to understand how much the model is confident about its predictions when there are multiple objects? So, let's say that a model achieves 100%. Then let's say that it predicts that in the first image the class is "bicycle", but there's clearly also a human there too. $\endgroup$
    – nbro
    Dec 24, 2021 at 10:54
  • $\begingroup$ Maybe another model does not achieve 100% accuracy and says that in the first image there's a human rather than a bicycle, assuming that both of these classes are present in the dataset. Now, your question seems to be related to how the model learns the representations of bicycles and humans (in this case) and how far away are they. But, again, I still don't understand the type of answer you're looking for. $\endgroup$
    – nbro
    Dec 24, 2021 at 10:54
  • $\begingroup$ So, I see multiple potential questions here that are related to 1. the confidence of the model to predict exactly one class (when there are many), 2. the representations that the model learns when there are many objects in the image. Your last sentence "The actual quality can be the same, in fact." suggests that you think that if a model predicts there's a human or bicycle in the first image, the quality would be the same. Well, yes, in a way, this is true. $\endgroup$
    – nbro
    Dec 24, 2021 at 10:57

1 Answer 1

Reset to default
2
$\begingroup$

This question is studied in a recent research - Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels.

It is quite strange, that community has paid so little attention to this issue.

They have relabeled the original ImageNet with the help of crops and changed the task to the multilabel.

enter image description here

This strategy turned out to be quite beneficial and improved accuracy of ResNet50 on validation set. At that time 80.2% accuracy with ResNet50 was the best result reported.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .