I want to detect dataset bias, and for that, the first approach is to build a model that can recognize from which dataset belongs an image.
I am working with Python3, with limited computing and more than 25k images of 512x512. There is a huge class imbalance, so the easiest way that I thought was to set the class_weights parameter in the data generator method of keras. This method is not working, my model is over fitting.
I have tried different approaches, but my main problem is that I have limited memory. I want to avoid to have to open each image in arrays, or to move them into a single directory, so I'm sticking with the flow_from_dataset method.
I tried using kfold to create small batches, move them to a single directory at a time and predict setting the class weight, but my model is only predicting the majority class. I'd prefer to not use data augmentation because my limited computer.
What do you recommend me to do? How to balance a large image dataset, where the images are in many different directories?
I have a dataframe with the path to each image and its label.
Thanks a lot