I'm working on an image classification problem using a neural network. In the training data set, 90% of the samples fall into 10% of all categories, while 10% of the sample fall into the other 90% categories. So an example is not evenly distributed among all categories. If we assume this distribution reflects the real-world distribution, do I need to filter my dataset before training so that each category has a similar number of samples?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ 8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset $\endgroup$– SmallChessOct 17, 2017 at 5:59
-
$\begingroup$ Likely of interest, especially one particular link contained within $\endgroup$– DaveOct 24, 2022 at 19:47
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Yes. Skewed data is one of the biggest problems in AI applications. As you rightly identified, the real world distribution is skewed. Doing a random sampling results in one major issue of an uneven sampling (like in your case). Even worse could happen, all of the samples may fall into a single class and other classes may not even be recognized by your classifier. This is called as the class imbalance problem.
There are many ways to mitigate this problem. Few of them have been mentioned here. I'll summarize them for you:
- Sampling based mitigation (when you try to deal with skewed data)
- Cost function based mitigation (when you try to improve the classifier, in case skewed data cannot be avoided)
Sampling based mitigation can be done by oversampling from the minority class(es), or undersampling from the majority class(es) or using a combination of both.
Another fairer method to do this is by doing a stratified sampling. If class A has 1/6th, class B has 2/6th and class C has 3/6th of the total population, then, you should take 1/6th of the samples by random sampling from objects in class A, 2/6th from class B and 3/6th from class C. This way all of the sample may represent the population in the right proportion, and none of the classes will be missed.
In case this doesn't help (say, you dataset is small and sampling doesn't make sense) and the data is skewed too much), the cost function can be modified to make a more sensible classifier. Keep different costs for misclassification and/or correct classification. Misclassification could be given a higher penalty. This could help in reaching to a better classifier with a skewed data.
$\begingroup$
$\endgroup$
Definitely. NNs can learn the data that you teach to them. If you teach them biased, the network will be biased. As you mention, one solution is to reduce the data that you have. However, it is not the best approach as you will be losing the precious data. I would suggest to try data augmentation for remaining dataset to increase missing data type samples and have similar numbers for best, evenly distributed accuracy.
