I am trying to design a model based on LSTM cells to do time-series prediction. The ouput value is an integer in [0,13]. I have noticed that one-hot encoding it and using cross-entropy loss gives better results than MSE loss.

Here is my problem : no matter how deep I make the network or how many fully connected layers I add I always obtain pretty much the same behavior. Changing the optimizer also doesn't really help.

  1. The loss function quickly decreases then stagnates with a very high variance and never goes down again.
  2. The prediction seems to be offset around the value 9, I really do not understand why since I have one-hot encoded the input and the output.

Here is an example of a the results of a typical training phase, with the total loss : enter image description hereenter image description here

Do you have any tips/ideas as to how I could improve this or could have gone wrong ? I am a bit of a beginner in ML si I might have missed something. I can also include the code (in PyTorch) if necessary.

  • $\begingroup$ What is the size of your data? And the number of trainable parameters in your network? $\endgroup$
    – Sharan
    Feb 19, 2020 at 13:41
  • $\begingroup$ The dataset is quite small (about 200 time-series of 50 points each). The number of trainable parameter depends, as said, I have tried various number of layers and hidden sizes but it didn't change that much. $\endgroup$
    – Johncowk
    Feb 19, 2020 at 14:19

1 Answer 1


I found the issue, I should have done more unit testing. Upon computing the batch loss before backpropagation, one of the dimension of the "prediction" tensor was not corresonding to the "truth" tensor. The shape match but the content is not the one that was supposed to be. This is due to how the NLL loss is implemented in pytorch which I was not aware of ...


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .