4
Short answer
To select the proper dataset to construct, you should first figure out a metric to use to compare, and then select the dataset construction that gives the better metric. There is no single best metric, it depends on the task and your interpretation on what type of error is more important.
If you believe it is important that errors should not be ...
3
Yes. Skewed data is one of the biggest problems in AI applications. As you rightly identified, the real world distribution is skewed. Doing a random sampling results in one major issue of an uneven sampling (like in your case). Even worse could happen, all of the samples may fall into a single class and other classes may not even be recognized by your ...
2
@nbro pointed out the paper A systematic study of the class imbalance problem in convolutional neural networks, which tested class imbalance LeNet for MNIST, on a custom CNN for CIFAR-10, and on ResNet for ImageNet. The paper found that by artificially creating class imbalance on those data sets, the neural networks are significantly deteriorated. The ROC ...
2
You can use stratified cross-validation combined with an imbalanced learning technique applied to the training data. Stratification ensures that when you split your data into train and test, the ratio of frequencies between the classes will stay the same, and therefore the test data will always be "realistic".
However, when training a model (using ...
1
The test set should represents the "real" data distribution your model will tackle once deployed and used in real applications. So the quick answer is yes, the test data should be imbalanced, which comes also as a sort of forced choice for you considering the super small size of you dataset.
You want to keep in mind though that this makes a bit ...
1
You can always adjust class weights accordingly. I know the reference is not for image data but it shouldn't matter if you are doing classification. Here is another answer more direct to the point.
1
One thing to try first is Focal Loss. This particular loss works well for classification or object detection where your dataset is unbalanced and contains many classes. In short, the loss suppresses highly confident predictions and gives the model more room to learn from other less confident classes.
You can read this blog to have more intuition about focal ...
1
The paper A systematic study of the class imbalance problem
in convolutional neural networks is a great overview on class imbalance approaches. Section 2 summarizes various methods commonly used. They categorize "Adding Class Weights for an imbalanced dataset" under the technique "Cost sensitive learning":
Cost sensitive learning. This ...
1
In my experience, GANs work really well for the scenario of semi-supervised learning, where you don't necessarily have labels for all your class $B$ data, but you do have a balanced dataset. In my (limited) experience, you do have to have a balanced (in numbers) set of $A$ and $B$ objects, even though you are not sure of the labels.
And yes, GANs can overfit ...
1
Definitely. NNs can learn the data that you teach to them. If you teach them biased, the network will be biased. As you mention, one solution is to reduce the data that you have. However, it is not the best approach as you will be losing the precious data. I would suggest to try data augmentation for remaining dataset to increase missing data type samples ...
Only top voted, non community-wiki answers of a minimum length are eligible