I trained a CNN using Keras in R to multi-dimensional image data for image classification of five classes. I realized that each run (I retrained the network on the same data for ten times), although I use the same data, my validation accuracy varies very much. On some runs, I get 48% accuracy and on some runs I get 70% accuracy. What are reasons which could lead to this unstable behaviour? I have a small amount of data, could this be the reason? If yes, can I fix this?
Since I don't know much about machine learning, I tried turning data augmentation on and off, using more or less data augmentation, using bigger or smaller batch sizes, using batch normalization after convolutional and fully-connected layers, and using a different kernel initializer. But none of these measures had much of an effect.
Edit: I have around 100 data points for validation and without data augmentation I have 400 data points for training.
Even more Edit: I originally have a dataset of roughly around 600 data points. I use 10-fold crossvalidation, the last fold is used for testing. I split the remaining 9 folds into 80% training data and 20% validation data. Originally, I wanted to do automatic hyperparameter optimization by grid search with the training & validation data, pick the best hyperparameters, retrain the models on training and validation data combined and then make a final test on the test data. However, I realized that each time I run one fold (also when I set a seed to reproduce the same training/validation-split and use fixed parameters), the validation accuracy varies a lot (out of curiosity I tested the performance on the test data too - it also varies that much). Because the accuracy varies so much, the model is almost always worse after training on the combined training and validation data. I guess a solution would be to just use the model produced while hyperparameter tuning (without refitting), but I don't think this is statistically correct and makes my results seem better than they really are. (here an image if I did not explain well)
These are some results of ten random example runs (same architecture, same data, same hyperparameters, same split):
> vali_loss loss loss loss loss loss loss loss loss loss loss 3.288659 3.773039 3.361197 3.027109 3.327728 3.033174 3.385153 3.236034 3.611694 3.411954 > vali_accs accuracy accuracy accuracy accuracy accuracy accuracy accuracy accuracy accuracy accuracy 0.5961539 0.5000000 0.5769231 0.6346154 0.5769231 0.6538461 0.6442308 0.5769231 0.5769231 0.4807692
Final Edit: I still have fluctuations, but I managed to decrease them to a somewhat more reasonable level. Instead of 10-fold CV I now use 5-fold CV so I have more data for accuracy calculation. Also, I now use a smaller learning rate, a smaller batch size and train for more epochs.