I am trying to design a model based on LSTM cells to do time-series prediction. The ouput value is an integer in
[0,13]. I have noticed that one-hot encoding it and using cross-entropy loss gives better results than MSE loss.
Here is my problem : no matter how deep I make the network or how many fully connected layers I add I always obtain pretty much the same behavior. Changing the optimizer also doesn't really help.
- The loss function quickly decreases then stagnates with a very high variance and never goes down again.
- The prediction seems to be offset around the value
9, I really do not understand why since I have one-hot encoded the input and the output.
Do you have any tips/ideas as to how I could improve this or could have gone wrong ? I am a bit of a beginner in ML si I might have missed something. I can also include the code (in PyTorch) if necessary.