I have coded a very basic LSTM with forget gates (no libraries used). I'm trying to predict $0.5*sin(t + N)$ given $0.5*sin(t)$ as an exercise.
I have tweaked the model, changing the output layer activation function, weight initialization, number of memory blocks/cells, etc. However, I still couldn't manage to correct the output.
The problem is that the output range is much smaller than desired, $[-0.2, 0.2]$ instead of $[-0.5, 0.5]$. The output also is slightly delayed, meaning it is predicting $sin(t + N - 1)$ for example.
Is there something that I'm missing?
As an example, for output layer activation function as a centered logistic from $(-1, 1)$, the validation output looks like
Training output looks like
Topology: 1 input layer, 1 hidden layer each with 5 memory blocks each with 1 cell, 1 output layer each with 1 regular neuron.
Weights: generated with normal distribution, from $[-1, 1]$
Output layer activation function used: logistic $[0, 1]$, centered logistic, tanh, ReLU, leaky ReLU, $f(x) = x$ (identity)