It looks like your network is overfitting, because the training loss carries on decreasing to zero even though validation loss levels off, and then starts to increase again.
I would guess that your network is essentially "memorising" the training examples because you're getting a near zero loss in training.
You could try:
- applying some form of regularisation
- reducing the number of neurons or layers
- collecting more training data to see if that helps.
Ideally you want to minimise validation loss, so epoch 20 was about the best point in your graph. Look into early stopping: this is an effective way to avoid overfitting too much.
When the learning rate is too high, you often see the loss 'jumping around' quite rapidly, and when the learning rate is too low, the graph barely changes at all. You could try fixing the learning rate closer to what it is around epoch 10 and see if this is "the sweet spot"; it's mostly a game of experimenting to see which works best.