I am working on an image data-set. As you may have guessed it is imbalanced data. I have 'Class A, 19,000 images' and 'Class B, 2,876 images'.
So I did an undersampling by removing randomly from the majority class till it becomes equal to the minority class.
On doing this I am loosing lot of information from those 19000 images which I could get. So I do an oversampling of minority class, by simply copying the 2,876 images again and again.
Is this undersampling method correct, will it effect my accuracy? I trained an Inceptionv4 model using this oversampled data and it is not at all stable and I am getting poor accuracy.
What should be my strategy ?