2
$\begingroup$

When using k-fold cross-validation in a deep learning problem, after you have computed your hyper-parameters, how do you decide how long to train your final model? My understanding is that, after the hyperparameters are selected, you train your model one more time on the entire set of data, but it's not clear to me when you decide to stop training.

$\endgroup$
2
$\begingroup$

Short answer: training "duration" or number of epochs/updates should be cross-validated too: you want to early-stop your training to prevent overfitting.

Longer answer:

Think of accuracy on the validation set as an estimate of accuracy on future data, given the value of some hyperparameter. In this case, the hyperparameter of interest is the number of training epochs. So: for each CV fold, train the network (e.g. up to some maximum number of epochs). After each epoch, record accuracy on the validation set. Compute the average validation set accuracy (across CV folds) for each number of epochs. Choose the number of epochs that maximizes this value.

https://stats.stackexchange.com/questions/298084/k-fold-cross-validation-for-choosing-number-of-epochs

| improve this answer | |
$\endgroup$
  • $\begingroup$ I have some problems with this but I'm having a hard time articulating them. I think at its core given the nature of SGD I don't completely trust this to be correct. $\endgroup$ – FourierFlux Apr 9 at 0:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.