CNN High Variance across multiple trained models, what does it mean?

Question

Background:

I have a 2D CNN model that I am applying to a regression task with some uniquely extracted spectrograms. The specifics of the data set are mostly irrelevant and very domain specific so I won't go into detail, but it is essentially just image classification with a MSE loss function for each label and a unique image of 100x4000. When I re-train the model from scratch multiple times then provide it my testing data set, it has predictions that vary significantly across each iteration and thus a high variance. Supposedly the only difference between a trained model versus another would be the random initialization of weights and the random train/validation split. I feel that the train/validation split has been ruled out by when I've done k-fold cross validation and my model has done very good for all segments of my train/validation splits and acquired good results for the validation in each split. But these same models persist to have high variance in the test data set.

Question:

If I am seeing a high variance for the predictions from my trained model across multiple different runs of re-training, what do I attack first to reduce my variance on my predictions for my test data set?

I've found many articles talking about bias and variance in the data set but not as much criticism directed towards model design. What things can I explore in my dataset or model design, and/or tools I can use to strengthen my model? Does my model need to be bigger/smaller?

Ideas/Solutions: A few Ideas I'd like to acquire some criticism for.

1. Regularization applied to model such as L1/L2 Regularization, dropout, or early stopping.
2. Data augmentation applied to dataset (inconveniently not an option right now, but in a more general scenario it could be).
3. Bigger or smaller model?
4. Is the random initialization of weight actually very important? Maybe train multiple models and take the average of their collective answers to get the best prediction on real world data (test set).

Personal Note: I have had experience with all these items before on other projects and with personal projects and have some moderate confidence justifying regularization and data augmentation. However, I lack some perspective as to any other tools that might be useful to explore the cause of model variance. I wanted to ask this question here to start a discussion in a general sense of this problem.

Cheers

EDIT: CLARIFICATION. When I say 'variance' I mean specifically variance across models, not variance of predictions from 1 trained model across the test set. Example: Instead lets say I am trying to predict a value somewhere between 1 and 4 (expected_val=3). I train 10 models to do this and 4 of the models accurately predict 3 with a VERY low standard deviation across all the test set samples. Thus a low variance and high accuracy/precision for these 4 models. But the other 6 models predict wildly and some predict 1 very confidently every time and the others could be 4. And I've even had models that predicted negative values even though I have NO training or testing samples that have negative labels.

Answer 1

Firstly if you're doing spectrogram classification, you'd probably want to use a loss function like cross entropy. That would give you less "high-variance" like results when you're calculating your evaluation metrics.

About your ideas:

• Dropout and early stopping would be enough

• Data augmentation with speech is quite helpful, and there are a few libraries to help you with volume/speed/noise augmentation. Volume/speed augmentation is actually really important and you'd want to do it anyway for decent results in production. Speech processing is sometimes a bit tricky, in the sense that what we hear may not be what ends up in the spectrogram and parameters like noise, recording setup, encoding, accent end up causing a domain-mismatch (covariate shift)

• In general, a smaller model (less parameters) helps with high variance. Start with small and simple models first, make sure eval metrics look ok, and then try larger models.
• Random initialization doesn't really affect the results in a systematic way. You can indeed ensemble predictions of models trained with different initialization; that does help with variance but don't expect a lot of difference. The gain is not worth the computation cost for models applied in practice (ie not kaggle)

Other ideas:

• Check that you shuffle data properly in train/test, and between folds
• Do some error analysis on the mis-predicted test samples, do they have something in common. Check the label distribution in your dev/test set.
• If the test set was captured in a process different from your dev set (total blind), don't expect a robust performance. In general, speech recognition is really data hungry with a decent "generic* model needing hundreds of hours of recording from different domains (tv, news, movies, telephone, lecture, echo, etc.).