1
$\begingroup$

I have a deep learning configuration in which I obtain good results on the validation set but even better results in the training set. From my understanding this means that there is overfitting to some extent. What does this mean in practice? Does it mean that my model is not good and that I should not use it? If I decrease the gap between the validation and training accuracy (decreasing the overfitting) but at the same time decrease the validation accuracy, which of the two models is better?

Below are some images to illustrate the two situations outlined previously:

High validation accuracy and even higher training accuracy

Smaller gap between validation and training accuracy

Improve this question
$\endgroup$
2
  • $\begingroup$ Higher validation is almost always better. If you think about it, the best network in regards to keeping the difference between training and validation is one that's terrible, because it will perform equally bad on both. If you're validation accuracy is increasing, you can be confident that if the network is over-fitting to the training data, it isn't to the point where it impacts performance. There will always be some level of over-fitting, but if validation accuracy continues to increase, that's ok. $\endgroup$
    – Recessive
    Sep 22, 2020 at 7:26
  • $\begingroup$ This is not overfitting, but the (always expected, in some degree) generalization gap. $\endgroup$
    – desertnaut
    Sep 24, 2020 at 16:24

1 Answer 1

Reset to default
1
$\begingroup$

Validation results will almost never be as good as training results; that's just natural. As long as they are not too different, you should be fine. What "too different" means depends on the particular data set and model you're using.

If you plot the curves for varying parameter values, when the training error keeps going down but the validation error starts going up again, that's when you know there is overfitting. In your second graph, after 14 epochs, we might see the start of overfitting. If you continue this until 20 epochs or so, it should be even more clear. I would guess that 12 is probably a good value for the number of epochs for that problem.

In the first graph, we don't see that happening yet. The model might not be well-suited (the gap between training and validation results is a bit larger) but that can also be because of too little data, or other factors. Perhaps that's just the best you can do; there might be noise in the data or something.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .