What happens if I train a network for more epochs, without using early stopping?

I have a question about training a neural network for more epochs even after the network has converged without using early stopping criterion.

Consider the MNIST dataset and a LeNet 300-100-10 dense fully-connected architecture, where I have 2 hidden layers having 300 and 100 neurons and an output layer having 10 neurons.

Now, usually, this network takes about 9-11 epochs to train and have a validation accuracy of around 98%.

What happens if I train this network for 25 or 30 epochs, without using early stopping criterion?

• Is something preventing you from doing it and observing the results? – Neil Slater Mar 5 at 12:18
• For LeNet 300-100-10 network, no. But for VGG-19, yes, since my GPU isn't powerful enough, and hence my question – Arun Mar 5 at 12:19
• Why are you asking about the LeNet network on MNIST then? Have you tried with the LeNet network on MNIST? If so, what happened? Are you asking whether you would get the same effect with VGG-19 on ImageNet? – Neil Slater Mar 5 at 12:24

 class val(tf.keras.callbacks.Callback):
# functions in this class adjust the learning rate
lowest_loss=np.inf
lowest_trloss=np.inf
best_weights=model.get_weights()
lr=float(tf.keras.backend.get_value(model.optimizer.lr))
epoch=0
highest_acc=0

def __init__(self):
super(val, self).__init__()
self.lowest_loss=np.inf
self.lowest_trloss=np.inf
self.best_weights=model.get_weights()
self.lr=float(tf.keras.backend.get_value(model.optimizer.lr))
self.epoch=0
self.highest_acc=0

def on_epoch_end(self, epoch, logs=None):
val.lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
val.epoch=val.epoch +1
v_loss=logs.get('val_loss')
v_acc=logs.get('accuracy')
loss=logs.get('loss')
if loss<val.lowest_trloss:
val.lowest_trloss=loss
if v_acc<.90:
val.best_weights=model.get_weights()
if v_acc<=.95 and loss>val.lowest_trloss:
lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
ratio=val.lowest_trloss/loss  # add a factor to lr reduction
new_lr=lr * .7 * ratio
tf.keras.backend.set_value(model.optimizer.lr, new_lr)
msg='{0}\n current training loss {1:7.5f}  is above lowest training loss of {2:7.5f}, reducing lr to {3:11.9f}{4}'
print(msg.format(Cyellow, loss, val.lowest_trloss, new_lr,Cend))
if val.lowest_loss > v_loss:
msg='{0}\n validation loss improved,saving weights with validation loss= {1:7.4f}\n{2}'
print(msg.format(Cgreen, v_loss, Cend))
val.lowest_loss=v_loss
val.best_weights=model.get_weights()

else:
if v_acc>.95 and val.lowest_loss<v_loss:
# reduce learning rate based on validation loss> val.best_loss
lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
ratio=val.lowest_loss/v_loss  # add a factor to lr reduction
new_lr=lr * .7 * ratio
tf.keras.backend.set_value(model.optimizer.lr, new_lr)
msg='{0}\n current loss {1:7.4f} exceeds lowest loss of {2:7.4f}, reducing lr to {3:11.9f}{4}'
print(msg.format(Cyellow, v_loss, val.lowest_loss, new_lr,Cend))

`

Training a neural network for "too many" epochs than needed without using early stopping criterion leads to overfitting, where your model's ability to generalize decreases.