I have coded a very basic LSTM with forget gates (no libraries used). I'm trying to predict $0.5*sin(t + N)$ given $0.5*sin(t)$ as an exercise.

I have tweaked the model, changing the output layer activation function, weight initialization, number of memory blocks/cells, etc. However, I still couldn't manage to correct the output.

The problem is that the output range is much smaller than desired, $[-0.2, 0.2]$ instead of $[-0.5, 0.5]$. The output also is slightly delayed, meaning it is predicting $sin(t + N - 1)$ for example.

Is there something that I'm missing?

As an example, for output layer activation function as a centered logistic from $(-1, 1)$, the validation output looks like

Validation output

Training output looks like

Training output

Additional information:

  • Topology: 1 input layer, 1 hidden layer each with 5 memory blocks each with 1 cell, 1 output layer each with 1 regular neuron.

  • Alpha: 1

  • Weights: generated with normal distribution, from $[-1, 1]$

  • Output layer activation function used: logistic $[0, 1]$, centered logistic, tanh, ReLU, leaky ReLU, $f(x) = x$ (identity)

  • $\begingroup$ can you provide this question with your code to help us find the problem? $\endgroup$ – malioboro Mar 4 '19 at 6:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.