Running for "to many" epochs can indeed lead to over fitting. You should look at the validation loss. If on AVERAGE it continues to decrease then you are not yet over fitting.
You may be tempted to run more epochs in hopes your loss will decrease but unless you adjust your learning rate dynamically at some point you won't get any improvement.
If you use KERAS it has a useful callback ReduceLROnPlateau. Documentation is at https://keras.io/callbacks/
This allows you to monitor a metric (typically validation loss) and to adjust the learning rate by a user defined factor(parameter factor) if the metric you are monitoring fails to improve after a certain number of consecutive epochs(parameter patience). You can think of the training process as travelling down a valley in N space(N being the number of trainable parameters). As you descend towards a minimum they valley gets narrower. If you do not lower the learning rate you will reach a point where you can not descend any further.Now you could use a very small learning rate to begin with but then you will have to train for a lot more epochs. One problem with adjusting the learning rate just on the validation loss is that in the early training epochs validation loss often does not track with training accuracy and it could cause the learning rate to be decreased prematurely.
I wrote a custom callback which initially monitors training loss and adjust the learning rate based on that metric. Once the training accuracy reaches 95% it switches to monitoring validation loss and adjusts the learning rate based on that. It saves the model weights for the lowest validation loss in the variable val.best_weights. After training load these weights into your model to make predictions. Code is below if you are interested. When you compile your model just add 'val' to the callback list.
class val(tf.keras.callbacks.Callback):
# functions in this class adjust the learning rate
lowest_loss=np.inf
lowest_trloss=np.inf
best_weights=model.get_weights()
lr=float(tf.keras.backend.get_value(model.optimizer.lr))
epoch=0
highest_acc=0
def __init__(self):
super(val, self).__init__()
self.lowest_loss=np.inf
self.lowest_trloss=np.inf
self.best_weights=model.get_weights()
self.lr=float(tf.keras.backend.get_value(model.optimizer.lr))
self.epoch=0
self.highest_acc=0
def on_epoch_end(self, epoch, logs=None):
val.lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
val.epoch=val.epoch +1
v_loss=logs.get('val_loss')
v_acc=logs.get('accuracy')
loss=logs.get('loss')
if loss<val.lowest_trloss:
val.lowest_trloss=loss
if v_acc<.90:
val.best_weights=model.get_weights()
if v_acc<=.95 and loss>val.lowest_trloss:
lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
ratio=val.lowest_trloss/loss # add a factor to lr reduction
new_lr=lr * .7 * ratio
tf.keras.backend.set_value(model.optimizer.lr, new_lr)
msg='{0}\n current training loss {1:7.5f} is above lowest training loss of {2:7.5f}, reducing lr to {3:11.9f}{4}'
print(msg.format(Cyellow, loss, val.lowest_trloss, new_lr,Cend))
if val.lowest_loss > v_loss:
msg='{0}\n validation loss improved,saving weights with validation loss= {1:7.4f}\n{2}'
print(msg.format(Cgreen, v_loss, Cend))
val.lowest_loss=v_loss
val.best_weights=model.get_weights()
else:
if v_acc>.95 and val.lowest_loss<v_loss:
# reduce learning rate based on validation loss> val.best_loss
lr=float(tf.keras.backend.get_value(self.model.optimizer.lr))
ratio=val.lowest_loss/v_loss # add a factor to lr reduction
new_lr=lr * .7 * ratio
tf.keras.backend.set_value(model.optimizer.lr, new_lr)
msg='{0}\n current loss {1:7.4f} exceeds lowest loss of {2:7.4f}, reducing lr to {3:11.9f}{4}'
print(msg.format(Cyellow, v_loss, val.lowest_loss, new_lr,Cend))```
