I trained a CNN using Keras in R to multi-dimensional image data for image classification of five classes. I realized that each run (I retrained the network on the same data for ten times), although I use the same data, my validation accuracy varies very much. On some runs, I get 48% accuracy and on some runs I get 70% accuracy. What are reasons which could lead to this unstable behaviour? I have a small amount of data, could this be the reason? If yes, can I fix this?

Since I don't know much about machine learning, I tried turning data augmentation on and off, using more or less data augmentation, using bigger or smaller batch sizes, using batch normalization after convolutional and fully-connected layers, and using a different kernel initializer. But none of these measures had much of an effect.

Edit: I have around 100 data points for validation and without data augmentation I have 400 data points for training.

Even more Edit: I originally have a dataset of roughly around 600 data points. I use 10-fold crossvalidation, the last fold is used for testing. I split the remaining 9 folds into 80% training data and 20% validation data. Originally, I wanted to do automatic hyperparameter optimization by grid search with the training & validation data, pick the best hyperparameters, retrain the models on training and validation data combined and then make a final test on the test data. However, I realized that each time I run one fold (also when I set a seed to reproduce the same training/validation-split and use fixed parameters), the validation accuracy varies a lot (out of curiosity I tested the performance on the test data too - it also varies that much). Because the accuracy varies so much, the model is almost always worse after training on the combined training and validation data. I guess a solution would be to just use the model produced while hyperparameter tuning (without refitting), but I don't think this is statistically correct and makes my results seem better than they really are. (here an image if I did not explain well)

These are some results of ten random example runs (same architecture, same data, same hyperparameters, same split):

> vali_loss
    loss     loss     loss     loss     loss     loss     loss     loss     loss     loss 
3.288659 3.773039 3.361197 3.027109 3.327728 3.033174 3.385153 3.236034 3.611694 3.411954 
> vali_accs
 accuracy  accuracy  accuracy  accuracy  accuracy  accuracy  accuracy  accuracy  accuracy  accuracy 
0.5961539 0.5000000 0.5769231 0.6346154 0.5769231 0.6538461 0.6442308 0.5769231 0.5769231 0.4807692 

Final Edit: I still have fluctuations, but I managed to decrease them to a somewhat more reasonable level. Instead of 10-fold CV I now use 5-fold CV so I have more data for accuracy calculation. Also, I now use a smaller learning rate, a smaller batch size and train for more epochs.

  • $\begingroup$ How varied are your image classes, what sizes of image, and are they natural images (e.g. photographs) or something more constructed (e.g. document scans, x-rays of same product with/without faults)? This seems like a good candidate for transfer learning. $\endgroup$ Jul 9 at 18:23
  • $\begingroup$ I don't know what you mean by varied, but I have 25x25x13 images. There are RGB bands included, but also a big number of abstract bands. The images are clips from geospatial rasters, so each image shows a unique location. Do you think my problem ist the small data set? I am very confused why accuracy fluctuates that much. If the model is so bad, I would expect consistently bad accuracies close to a random accuracy. Instead, it fluctuates around 55%... $\endgroup$
    – user48488
    Jul 9 at 18:49
  • 1
    $\begingroup$ The problem is the small dataset, but that is not necessarily causing a large variation in accuracy (there will be some of course). It is mainly causing you problems with measuring your accuracy. $\endgroup$ Jul 9 at 18:56
  • $\begingroup$ Does that mean that you would expect the validation losses of the model to be quite similar? $\endgroup$
    – user48488
    Jul 9 at 19:06
  • 1
    $\begingroup$ But in short, yes, validation is a form of measurement. Validation merics are more trustworthy than training ones for assessing performance on unseen data, but they are still subject to issues of sample size, and metrics such as accuracy on a random sample from a population are effectively random variables being measured. $\endgroup$ Jul 9 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy