I'm trying to teach an LSTM to predict the next values in 3 related series. (Financial data)

Unfortunately, it looks like I made some basic mistake and this network never gets past just returning constant data. This is very confusing, because I've checked all the basics and even the most basic approaches like "return the last value" or "repeat the last change" give lower loss than the constant.

I'm using the most basic LSTM model in pytorch lightning. Single LSTM with between 5 and 100 hidden values and between 1 and 10 layers (doesn't change the result).

self.model = nn.Sequential(
    nn.LSTM(len(columns), 50, 5, batch_first=True),

My output is the last hidden value:

_, (output, _) = self(inputs)
loss = torch.nn.functional.mse_loss(output[-1,:,:3], target.squeeze())

I'm using AdamW (starting at the default rate) with ReduceLROnPlateau (based on validation loss). I'm using log(price) rescaled to around 0..1 range. I've added extra tech indicators also scaled to that range. I've tried using difference-to-previous instead of the values themselves.

Every single time, I get fairly fast convergence to just returning a constant. What stupid thing am I missing here? What should I be checking in a situation like this?

  • $\begingroup$ Please, add some example of " past just returning constant data.". Also as Kulin Patel asked in the suggestion section, it would be helpful your dataset information with your training hyper parameter like epoch, and etc. By the way, where do you get the idea of using Long Short Term Memory Network Model (any reference)? $\endgroup$
    – Cloud Cho
    Commented Jun 13 at 0:33


You must log in to answer this question.