This is more of a conceptual question that I am asking here. I am using an encoder-decoder model with Bidirectional LSTMs used as a time series regression problem. I train it with a sufficiently low learning rate (1e-5) and an early-stop callback with a patience of 25 epochs, meaning that training of the model stops automatically when the validation loss is stuck at a constant trend for 25 extra epochs (My validation loss does decrease significantly from 1 to around 0.06). This brings my total trained epochs to around 50-80. I am sure that I am using sufficient layers of LSTMs and the TimeDistributed Dense layers. Despite this, I am getting some errors in prediction.

My question to you with respect to the above scenario is: Does the low and constant validation loss mean that the model cannot give a better prediction performance if trained for further epochs? In other words, is it possible that the model still needs more epochs to learn to give better model prediction performance, even though the validation loss has decreased as much as it could?

  • $\begingroup$ The model may benefit slightly from further epochs, but more than likely you are just going to over fit the training data. I noticed with your method here that you don't seem to be using attention. Attention is a really powerful tool specifically created for this exact model in the seq2seq context. If you haven't implemented attention, you will probably achieve a better accuracy once you have. Especially if the input length is particularly long. $\endgroup$ – Recessive Aug 20 '20 at 3:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.