How to interpret the following learning curves?
Background: The accuracy starts at 50%, because the network has a binary output (0 or 1). I chose an exponentially decreasing learning rate of the optimizer - I believe that this the reason why the network starts learning after 10 epochs or so.
lr_schedule = keras.optimizers.schedules.ExponentialDecay(**h_p["optimizer"]["adam"])
optimizer = keras.optimizers.Adam(learning_rate=lr_schedule)
h_p["Compile"]["optimizer"] = optimizer
It looks like your network is overfitting, because the training loss carries on decreasing to zero even though validation loss levels off, and then starts to increase again.
I would guess that your network is essentially "memorising" the training examples because you're getting a near zero loss in training.
You could try:
Ideally you want to minimise validation loss, so epoch 20 was about the best point in your graph. Look into early stopping: this is an effective way to avoid overfitting too much.
When the learning rate is too high, you often see the loss 'jumping around' quite rapidly, and when the learning rate is too low, the graph barely changes at all. You could try fixing the learning rate closer to what it is around epoch 10 and see if this is "the sweet spot"; it's mostly a game of experimenting to see which works best.
