1
$\begingroup$

I have a question about training a neural network for more epochs even after the network has converged without using early stopping criterion.

Consider the MNIST dataset and a LeNet 300-100-10 dense fully-connected architecture, where I have 2 hidden layers having 300 and 100 neurons and an output layer having 10 neurons.

Now, usually, this network takes about 9-11 epochs to train and have a validation accuracy of around 98%.

What happens if I train this network for 25 or 30 epochs, without using early stopping criterion?

$\endgroup$
  • $\begingroup$ Is something preventing you from doing it and observing the results? $\endgroup$ – Neil Slater Mar 5 at 12:18
  • $\begingroup$ For LeNet 300-100-10 network, no. But for VGG-19, yes, since my GPU isn't powerful enough, and hence my question $\endgroup$ – Arun Mar 5 at 12:19
  • $\begingroup$ Why are you asking about the LeNet network on MNIST then? Have you tried with the LeNet network on MNIST? If so, what happened? Are you asking whether you would get the same effect with VGG-19 on ImageNet? $\endgroup$ – Neil Slater Mar 5 at 12:24
1
$\begingroup$

Running for "to many" epochs can indeed lead to over fitting. You should look at the validation loss. If on AVERAGE it continues to decrease then you are not yet over fitting. You may be tempted to run more epochs in hopes your loss will decrease but unless you adjust your learning rate dynamically at some point you won't get any improvement. If you use KERAS it has a useful callback ReduceLROnPlateau. Documentation is at https://keras.io/callbacks/ This allows you to monitor a metric (typically validation loss) and to adjust the learning rate by a user defined factor(parameter factor) if the metric you are monitoring fails to improve after a certain number of consecutive epochs(parameter patience). You can think of the training process as travelling down a valley in N space(N being the number of trainable parameters). As you descend towards a minimum they valley gets narrower. If you do not lower the learning rate you will reach a point where you can not descend any further.Now you could use a very small learning rate to begin with but then you will have to train for a lot more epochs. One problem with adjusting the learning rate just on the validation loss is that in the early training epochs validation loss often does not track with training accuracy and it could cause the learning rate to be decreased prematurely. I wrote a custom callback which initially monitors training loss and adjust the learning rate based on that metric. Once the training accuracy reaches 95% it switches to monitoring validation loss and adjusts the learning rate based on that. It saves the model weights for the lowest validation loss in the variable val.best_weights. After training load these weights into your model to make predictions. Code is below if you are interested. When you compile your model just add 'val' to the callback list.

 class val(tf.keras.callbacks.Callback):
        # functions in this class adjust the learning rate 
        lowest_loss=np.inf
        lowest_trloss=np.inf
        best_weights=model.get_weights()
        lr=float(tf.keras.backend.get_value(model.optimizer.lr))
        epoch=0
        highest_acc=0

        def __init__(self):
            super(val, self).__init__()
            self.lowest_loss=np.inf
            self.lowest_trloss=np.inf
            self.best_weights=model.get_weights()
            self.lr=float(tf.keras.backend.get_value(model.optimizer.lr))
            self.epoch=0
            self.highest_acc=0


        def on_epoch_end(self, epoch, logs=None):             
            val.lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
            val.epoch=val.epoch +1            
            v_loss=logs.get('val_loss')
            v_acc=logs.get('accuracy')
            loss=logs.get('loss')
            if loss<val.lowest_trloss:
                val.lowest_trloss=loss
                if v_acc<.90:
                    val.best_weights=model.get_weights()
            if v_acc<=.95 and loss>val.lowest_trloss:
                lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
                ratio=val.lowest_trloss/loss  # add a factor to lr reduction
                new_lr=lr * .7 * ratio
                tf.keras.backend.set_value(model.optimizer.lr, new_lr)
                msg='{0}\n current training loss {1:7.5f}  is above lowest training loss of {2:7.5f}, reducing lr to {3:11.9f}{4}'
                print(msg.format(Cyellow, loss, val.lowest_trloss, new_lr,Cend))   
            if val.lowest_loss > v_loss:
                msg='{0}\n validation loss improved,saving weights with validation loss= {1:7.4f}\n{2}'
                print(msg.format(Cgreen, v_loss, Cend))
                val.lowest_loss=v_loss
                val.best_weights=model.get_weights()

            else:
                 if v_acc>.95 and val.lowest_loss<v_loss:
                        # reduce learning rate based on validation loss> val.best_loss
                        lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
                        ratio=val.lowest_loss/v_loss  # add a factor to lr reduction
                        new_lr=lr * .7 * ratio
                        tf.keras.backend.set_value(model.optimizer.lr, new_lr)
                        msg='{0}\n current loss {1:7.4f} exceeds lowest loss of {2:7.4f}, reducing lr to {3:11.9f}{4}'
                        print(msg.format(Cyellow, v_loss, val.lowest_loss, new_lr,Cend))```


| improve this answer | |
$\endgroup$
1
$\begingroup$

Training a neural network for "too many" epochs than needed without using early stopping criterion leads to overfitting, where your model's ability to generalize decreases.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.