I am using Keras to build a CNN model to classify spectograms and using the following layers:
Conv2D 32 filters, size (5,5), relu MaxPooling2D (2,2) Conv2D 32 filters, size (5,5), relu MaxPooling2D (2,2) Conv2D 64 filters, size (5,5), relu MaxPooling2D (2,2) Flatten Dense 64 , relu Dense 32 , relu Dense 5 (softmax)
With this model I was able to comfortably achieve more than 97% training accuracy and around 84% validation accuracy. Unfortunately, I have very few data samples (slightly more than 1200), which is my main suspicion on why the model is overfitting.
As I am not able to get more data samples, I opted to use SpecAugmentation. I used to get all my classes (5) to 1000 samples each, meaning I generated around 3800 samples.
After I fit the model to this augmented data I got more than 97% validation data, which I found very strange. I am thinking that I am introducing bias because I am performing the data augmentation before splitting into training and validation sets, which may lead to having the same sample in both sets (obviously they are not 100% equal but one is derived from the other).
Should I only be performing data augmentation after splitting? If I use data augmentation only on the training set will I not be overfitting to the training set as well, as it will see very similar samples several times?
What other techniques would you recommend for data augmentation besides SpecAugmentation?