1
$\begingroup$

As shown below, my deep neural network is overfitting : Cross-entropy loss Accuracy

where the blue lines is the metrics obtained with training set and red lines with validation set

Is there anything I can infer from the fact that the accuracy on the training sets is really high (almost 1) ?

From what I understand, it means that the complexity of my model is enough / too big. But does it means my model could theoretically reach such a score on validation set with same dataset and appropriate hyperparameters ? With same hyperparameters but bigger dataset ?

My question is not how to avoid overfitting.

$\endgroup$
  • $\begingroup$ "But does it means my model could theoretically reach such a score on validation set with same dataset and appropriate hyperparameters ? With same hyperparameters but bigger dataset ?" It's a question about how to avoid overfitting ;) $\endgroup$ – Jérémy Blain Oct 10 '18 at 8:37
  • $\begingroup$ @JérémyBlain Sorry for my poor english, I'm not native. My sentence might not convey the sens I wished. I was asking some interpretation based on the (overfitting) training accuracy. I know the methods to avoid overfitting. I was more asking about experience with overfitting, if it is possible to reach such a good accuracy without applying all the counter-overfitting methods. $\endgroup$ – Astariul Oct 11 '18 at 6:14
  • 1
    $\begingroup$ See chapter 2.5 of Analysis and Optimization of Convolutional Neural Network Architectures $\endgroup$ – Martin Thoma Oct 13 '18 at 8:57
0
$\begingroup$

It doesn't tell you very much, to be honest. It does mean that (assuming your training and validation distributions are similar) your model could get the same results on your validation set should you train on that, but that would still be overfitting.

Really, the only useful thing overfitting tells you is that you don't have enough regularisation.

$\endgroup$
  • $\begingroup$ overfitting tells you that your model don't generalize well. Regularization is not the only method to reduce overfitting (think of a bigger dataset) $\endgroup$ – Jérémy Blain Oct 10 '18 at 8:44
  • $\begingroup$ I would argue that using more data, augmenting data, reducing the size of the net, etc. are all forms of regularisation. $\endgroup$ – Omegastick Oct 10 '18 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.