Why is the validation loss less than the training loss, and what can be said about the effect of the learning rate?

Question

I have the following results I am trying to make sense of. I have attached the loss curves here for reference.

1. As you can see, the first issue is that the validation loss is lower than the training loss. I think this is due to using a pre-trained model with a high dropout rate (please correct me if I am wrong here).

2. As one can see, the mean_auc score is increasing consistently, and so it seems that the network is indeed learning something and the validation loss is also better behaved relatively.

3. The training loss is what bugs me a lot. It is not at all consistent and varies a lot. This is a naive question, but is this graph giving me any sort of information about the learning rate, etc, or am I in a situation wherein everything is incorrect essentially?

Any response would be really appreciated.

Answer 1

This is very difficult to tell with the information provided, but the phenomenon is something that I have encountered many times before. Sometimes this is not a bad thing, here are some possible considerations/explanations:

1. Data from the training set could be identical or leaking in to the validation set.
2. Using a high dropout rate can cause this as well as other generalization techniques.
3. The training set may contain outliers, contributing to higher loss and difficulty in learning.

Also, note that the there is not so much an issue with the training loss bouncing around as long as it decreases over time. You may want to try different optimizers as well to see if that changes this behavior.