2
$\begingroup$

I have a neural network where there are two hidden layers. Each hidden layer has 128 neurons. The input layer has 20 inputs, and the output layer has 3 outputs.

I have 1 million records of data. 80% is used to train the network, 20% is used for validation. I run the training for 100000 epochs.

I see that the neural network attains 100% accuracy on the training data after only 12000 epochs.

Should I stop training or continue until all 100000 epochs are complete? Please, explain why.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ You don't appear to have enough splits for normal best practice here. There should be a three-way split. How are you selecting between models, and following that how are you assessing accuracy in an unbiased fashion? $\endgroup$
    – Neil Slater
    Jul 6, 2021 at 6:20
  • 3
    $\begingroup$ As a rule of thumb, generally 100% training accuracy means you've overtrained. Training accuracy isn't what you should be looking at anyway, as mentioned in the other comments, you should be looking at validation accuracy. $\endgroup$
    – Recessive
    Jul 6, 2021 at 11:50

1 Answer 1

Reset to default
6
$\begingroup$

First of all, as mentioned by @Neil Slater in the comment - you need to have three splits into the train, validation and test set.

One sometimes disregards the difference between validation and test set. However they serve for different purposes. Here I would like to cite https://machinelearningmastery.com/difference-test-validation-datasets/ :

Validation Dataset: The sample of data used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyperparameters. The evaluation becomes more biased as skill on the validation dataset is incorporated into the model configuration.

Test Dataset: The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset.

Secondly, in order to understand what's happening plot jointly the train and validation loss. In case the performance on validation data becomes much worse, that on the training - It is better to terminate training, since it is the indication of overfitting.

A good practice is to use early stopping, there is an implementation of this callback in Tensorflow - https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping.

It a kind of regularization procedure https://en.wikipedia.org/wiki/Early_stopping.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Does anyone talk about "orders of validation". Ie, you have your order-0 validation (your training set), your order-1 validation (your validation set, which is used to determine what your model hyperparameters should be), then order-2 validation (your "test" dataset, which is not used to tune hyperparameters, but to determine if you screwed up), your order-3 validation (which you use only if your "test" dataset told you that your result was junk); the idea is that order-N validation is used to validate after previous validations where used as input into your model, which adds bias. $\endgroup$
    – Yakk
    Jul 6, 2021 at 15:19
  • $\begingroup$ @Yakk Maybe you're talking about k-fold cross-validation? $\endgroup$
    – nbro
    Jul 6, 2021 at 15:21
  • 1
    $\begingroup$ @nbro No, something different. Yakk is generalizing the difference between test and validation sets. The reason you need two is because your training loop has two levels of nesting. If you somehow needed, say, three more levels of nesting, you'd need 5 of those sets. $\endgroup$
    – user253751
    Jul 6, 2021 at 15:34
  • $\begingroup$ As far as I am aware, this idea is not common in ML, but it's possible that something along those lines has been applied. Typically, you use the same validation data for early stopping or/and hyper-parameter tuning. I agree that, if you used a certain validation dataset for hyper-parameter tuning, then selected a specific function using another validation dataset, you could be biased (and the bias would come from the choice of the specific hyper-parameters with the previous val. data, which, of course, also limits the specific function from the specific hypothesis class you can choose). $\endgroup$
    – nbro
    Jul 6, 2021 at 15:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .