5
$\begingroup$

Collecting and labeling training data for supervised learning tasks is incredibly time-consuming and costly.

For instance, let's say you wrote a script that went on Google images and got you 5000 pictures for each of 10 classes. You then use an unsupervised algorithm to cluster them. Then, you train a supervised algorithm using the labels from the scraper as ground truth. Obviously, your network will perform more poorly than one with perfectly labeled data, but is there a way to guesstimate how much?

Perhaps there are 50 mislabeled images in each class. That would most likely be better than 500 mislabeled images, but I'm wondering if there is a way to predict how much (even if it is by someone's rules of thumb or something like that).

Improve this question
$\endgroup$
1
  • $\begingroup$ There exists a multitude of work on dealing with label noisy data if you look into it. One approach that’s super easy that I find useful was relating to a paper that came out last year in creating a super small gold label classifier and then use that as a prior for the larger model $\endgroup$
    – mshlis
    Commented Aug 3, 2019 at 20:16

2 Answers 2

Reset to default
3
$\begingroup$

I think the crucial point here is what you precisely mean by mislabelled. Google's image classifier will likely do a 'pretty good' job of retrieving images with the given subject included, but how strict or lenient your class requisites are is quite important. For example, if one of your classes is 'dog' there may be hundreds of images procured from scraping that could display (examples off the top of my head, but you can get even more creative):

  • Ancient canine fossils
  • Wolves
  • Stuffed animal dogs
  • Hard mode dogs (i.e. partially occluded, variable lighting schemas, background color variation, intraclass/species variation)

Additionally, your computational tool will impact this. If you're using a neural network, the above issues can to some extent be accounted for, but a linear classifier would likely have difficulty adopting a broader/flexible view of your class.

Improve this answer
$\endgroup$
1
$\begingroup$

I will break it down for you in very simple words. The accuracy will drop down as you label them wrong. In simpler words- accuracy is directly proportional on how perfect the data is labelled. If you think about it, suppose you have 2 categories-cats and dogs, and you have a dataset of 10,000 pictures. Out of which 50 are wrongly labelled. The accuracy will less than the perfectly labelled but not that less since the neural network built will not be that bad. But suppose now you have 1000 wrongly labelled which is 1/10 of the dataset, then the NN will have more abrupt outcomes.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .