1
$\begingroup$

I am training a convLSTM with a dropout layer (with prob 0.5).

If I train over more than 5 epochs I notice that the network starts to overfit: my validation set loss becomes stationary while the train loss keeps going down with every epoch.

And if I train for 20 or more epochs the gap between the validation and train loss is quite substantial. At the same time precision-recall curve becomes much more stable (i.e. monotonic) if i train with a large number of epochs (e.g. 20). Why is that? Is this behaviour a common occurrence?

Improve this question
$\endgroup$
5
  • $\begingroup$ Precision-recall on training set or CV set? $\endgroup$ – DuttaA Apr 11 '19 at 14:04
  • $\begingroup$ on the validation set $\endgroup$ – hellmean Apr 11 '19 at 14:14
  • 1
    $\begingroup$ if validation loss does not bounce and start to get worse I would not call it overfitting...but what you are describing is common and not necessarily bad...the NN is memorizing the training examples, but if it is not impacting the validation loss adversely it might not be a bad thing...to get better validation loss you can try more/better data or regularization like L2. $\endgroup$ – j314erre Apr 11 '19 at 14:54
  • $\begingroup$ Is it monotonically increasing .. The precision recall? $\endgroup$ – DuttaA Apr 11 '19 at 15:57
  • $\begingroup$ yes, the precision-recall curve for the model trained with 20 epochs. When you train with less epochs it can have multiple dips (local minima). $\endgroup$ – hellmean Apr 12 '19 at 14:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.