3
$\begingroup$

I am training a modified VGG16 network for classification (adding 0.5 dropout after each of the last FC layers). In the following plot I am training for a small number of epochs as an example, and it shows the accuracy and loss curves of training process on both training and validation datasets. My training set size is $1725$, and $429$ for validation. Also I am training with weights=None

enter image description here

My question is about the validation curves, why do not they appear to be as smooth as the training ones? Is this normal during the training stage?

$\endgroup$
3
$\begingroup$

You are training your model on the train set and only validating your model on CV set, thus your weights are getting exclusively optimised according to the loss of Training Set (in a continuous manner) and thus always decreasing. We do not have such guarantees with the CV set, which is the entire purpose of Cross Validation in the first place. Ideally it gives you an unbiased measure of how well your model and its trained weights will perform in the real world. Thus even if it performs well in Training Set, the loss can still go up in CV set which is what you are seeing in your graph.

Speaking in layman terms, even if you do all the sums of a single exercise given in a book, your performance might not be the same in a model paper. You do another exercise from another book, there is no guarantee your previous concepts will stick and you may do even worse in the model paper. Same thing is happening here where exercises are your training set and model papers are to evaluate your learning.

$\endgroup$
  • $\begingroup$ thank you. I get from you answer that it is normal and validation performance is what counts after all. Thus I need to keep on modifying the model until I achieve higher validation accuracy. $\endgroup$ – norahik Apr 10 at 18:30
  • 1
    $\begingroup$ @norahik yes but the final test is the test set...After that you shouldn't modify your weights, it's what you report as your performance. So make sure you do perfect validation. $\endgroup$ – DuttaA Apr 10 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.