The specific problem I have is learning the relation $x^2$. I have an array of 0 through 19 (input values) and a target array of 0, 1, 4, 9, 16, 25, 36 and so on all the way up to $19^2$=361.
I have the following LSTM architecture:
1 input node 1 output node 32 hidden units
Now, interestingly, I accidentally trained my network wrong, in that I forgot to reverse the expected output list when training. So I trained the network for: $$0 \rightarrow 361 \\ 1 \rightarrow 324 \\ 2 \rightarrow 289 \\ 3 \rightarrow 256 \\ ... \\ 17 \rightarrow 4 \\ 18 \rightarrow 1 \\ 19 \rightarrow 0$$
Starting with a learning rate of 1 and halving it every 400 epochs, after 3000 epochs my error (which started somwhere in the millions) was 0.2.
However, when I went to correct this mistake, my error will hardly ever go beneath 100,000. Testing the network shows it does well in the low inputs, but once it starts to get to $16^2$ onwards, it really struggles to increase the output values past ~250.
I was just wondering if anyone has an explanation for this, as to why the LSTM struggles to learn to increase exponentially but seems to be able to learn to decrease exponentially just fine.