# time-series prediction : loss going down, then stagnates with very high variance

I am trying to design a model based on LSTM cells to do time-series prediction. The ouput value is an integer in [0,13]. I have noticed that one-hot encoding it and using cross-entropy loss gives better results than MSE loss.

Here is my problem : no matter how deep I make the network or how many fully connected layers I add I always obtain pretty much the same behavior. Changing the optimizer also doesn't really help.

1. The loss function quickly decreases then stagnates with a very high variance and never goes down again.
2. The prediction seems to be offset around the value 9, I really do not understand why since I have one-hot encoded the input and the output.

Here is an example of a the results of a typical training phase, with the total loss :

Do you have any tips/ideas as to how I could improve this or could have gone wrong ? I am a bit of a beginner in ML si I might have missed something. I can also include the code (in PyTorch) if necessary.

• What is the size of your data? And the number of trainable parameters in your network? – Sharan Feb 19 at 13:41
• The dataset is quite small (about 200 time-series of 50 points each). The number of trainable parameter depends, as said, I have tried various number of layers and hidden sizes but it didn't change that much. – Johncowk Feb 19 at 14:19