# What happens if I train a network for more epochs, without using early stopping?

I have a question about training a neural network for more epochs even after the network has converged without using early stopping criterion.

Consider the MNIST dataset and a LeNet 300-100-10 dense fully-connected architecture, where I have 2 hidden layers having 300 and 100 neurons and an output layer having 10 neurons.

Now, usually, this network takes about 9-11 epochs to train and have a validation accuracy of around 98%.

What happens if I train this network for 25 or 30 epochs, without using early stopping criterion?

• Is something preventing you from doing it and observing the results? – Neil Slater Mar 5 '20 at 12:18
• For LeNet 300-100-10 network, no. But for VGG-19, yes, since my GPU isn't powerful enough, and hence my question – Arun Mar 5 '20 at 12:19
• Why are you asking about the LeNet network on MNIST then? Have you tried with the LeNet network on MNIST? If so, what happened? Are you asking whether you would get the same effect with VGG-19 on ImageNet? – Neil Slater Mar 5 '20 at 12:24

## 2 Answers

Running for "to many" epochs can indeed lead to over fitting. You should look at the validation loss. If on AVERAGE it continues to decrease then you are not yet over fitting. You may be tempted to run more epochs in hopes your loss will decrease but unless you adjust your learning rate dynamically at some point you won't get any improvement. If you use KERAS it has a useful callback ReduceLROnPlateau. Documentation is at https://keras.io/callbacks/ This allows you to monitor a metric (typically validation loss) and to adjust the learning rate by a user defined factor(parameter factor) if the metric you are monitoring fails to improve after a certain number of consecutive epochs(parameter patience). You can think of the training process as travelling down a valley in N space(N being the number of trainable parameters). As you descend towards a minimum they valley gets narrower. If you do not lower the learning rate you will reach a point where you can not descend any further.Now you could use a very small learning rate to begin with but then you will have to train for a lot more epochs. One problem with adjusting the learning rate just on the validation loss is that in the early training epochs validation loss often does not track with training accuracy and it could cause the learning rate to be decreased prematurely. I wrote a custom callback which initially monitors training loss and adjust the learning rate based on that metric. Once the training accuracy reaches 95% it switches to monitoring validation loss and adjusts the learning rate based on that. It saves the model weights for the lowest validation loss in the variable val.best_weights. After training load these weights into your model to make predictions. Code is below if you are interested. When you compile your model just add 'val' to the callback list.

 class val(tf.keras.callbacks.Callback):
# functions in this class adjust the learning rate
lowest_loss=np.inf
lowest_trloss=np.inf
best_weights=model.get_weights()
lr=float(tf.keras.backend.get_value(model.optimizer.lr))
epoch=0
highest_acc=0

def __init__(self):
super(val, self).__init__()
self.lowest_loss=np.inf
self.lowest_trloss=np.inf
self.best_weights=model.get_weights()
self.lr=float(tf.keras.backend.get_value(model.optimizer.lr))
self.epoch=0
self.highest_acc=0

def on_epoch_end(self, epoch, logs=None):
val.lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
val.epoch=val.epoch +1
v_loss=logs.get('val_loss')
v_acc=logs.get('accuracy')
loss=logs.get('loss')
if loss<val.lowest_trloss:
val.lowest_trloss=loss
if v_acc<.90:
val.best_weights=model.get_weights()
if v_acc<=.95 and loss>val.lowest_trloss:
lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
ratio=val.lowest_trloss/loss  # add a factor to lr reduction
new_lr=lr * .7 * ratio
tf.keras.backend.set_value(model.optimizer.lr, new_lr)
msg='{0}\n current training loss {1:7.5f}  is above lowest training loss of {2:7.5f}, reducing lr to {3:11.9f}{4}'
print(msg.format(Cyellow, loss, val.lowest_trloss, new_lr,Cend))
if val.lowest_loss > v_loss:
msg='{0}\n validation loss improved,saving weights with validation loss= {1:7.4f}\n{2}'
print(msg.format(Cgreen, v_loss, Cend))
val.lowest_loss=v_loss
val.best_weights=model.get_weights()

else:
if v_acc>.95 and val.lowest_loss<v_loss:
# reduce learning rate based on validation loss> val.best_loss
lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
ratio=val.lowest_loss/v_loss  # add a factor to lr reduction
new_lr=lr * .7 * ratio
tf.keras.backend.set_value(model.optimizer.lr, new_lr)
msg='{0}\n current loss {1:7.4f} exceeds lowest loss of {2:7.4f}, reducing lr to {3:11.9f}{4}'
print(msg.format(Cyellow, v_loss, val.lowest_loss, new_lr,Cend))

`

Training a neural network for "too many" epochs than needed without using early stopping criterion leads to overfitting, where your model's ability to generalize decreases.