3
$\begingroup$

The specific problem I have is learning the relation $x^2$. I have an array of 0 through 19 (input values) and a target array of 0, 1, 4, 9, 16, 25, 36 and so on all the way up to $19^2$=361.

I have the following LSTM architecture:

1 input node
1 output node
32 hidden units

Now, interestingly, I accidentally trained my network wrong, in that I forgot to reverse the expected output list when training. So I trained the network for: $$0 \rightarrow 361 \\ 1 \rightarrow 324 \\ 2 \rightarrow 289 \\ 3 \rightarrow 256 \\ ... \\ 17 \rightarrow 4 \\ 18 \rightarrow 1 \\ 19 \rightarrow 0$$

Starting with a learning rate of 1 and halving it every 400 epochs, after 3000 epochs my error (which started somwhere in the millions) was 0.2.

However, when I went to correct this mistake, my error will hardly ever go beneath 100,000. Testing the network shows it does well in the low inputs, but once it starts to get to $16^2$ onwards, it really struggles to increase the output values past ~250.

I was just wondering if anyone has an explanation for this, as to why the LSTM struggles to learn to increase exponentially but seems to be able to learn to decrease exponentially just fine.

EDIT WITH CODE:

a = np.array([i for i in range(20)])
b = np.array([i**2 for i in range(20)])
np.random.seed(5)
ls = LSTM(1, 1, 32, learning_rate=1, regression=1)
# Input size = 1, output size = 1, hidden units = 32
if 1:
    for k in range(3000):
        total = 0
        for i in range(20):
            ls * a[i]
        for i in range(20):
            total += ls / b[i]
        if k % 400 == 0:
            ls.learning_rate *= 0.5
        print(k, ":", total)
        ls.update_params()
for i in a:
    print(i, ls*i)
#ls.save_state('1.trainstate')
for i in range(20,30):
    print(ls*i)

Note this code uses a class I wrote using numpy. If wanted I'll include this code as well it's just that is ~300 lines and I don't expect anyone to go through all that

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ I suspect that it is a problem with vanishing gradients, something like: If you feed in 19, you saturate the tanh in the LSTM, driving it to the regions where the gradient is almost 0, which is more hurtful when the target is 361 then when the target is 0 (since your output at the beginning is 0). But this is difficult to verify without having access to the model. Have you looked at the gradients in tensorboard? And could you post the mapping after training? I would also suggest to train the model again, but this time scale input and output to 0 to 1, and see if this observation still holds. $\endgroup$
    – Luca Thiede
    Jan 14, 2020 at 10:03
  • $\begingroup$ @LucaThiede That's a really good suggestion - it makes intuitive sense. I'll have to run the model again with scaled input/output. I'll try and create some form of visualisation of the gradients at each step $\endgroup$
    – Recessive
    Jan 17, 2020 at 1:35
  • 2
    $\begingroup$ @Recessive First LSTM shouldnt be used for this type of problem as its not temporal, second I ran this and got same loss for both ways (for me using MSE and same lr scheduler I get ~28000 loss), share your code and let us see if we can reproduce your behavior $\endgroup$
    – mshlis
    Jan 17, 2020 at 18:44
  • $\begingroup$ @mshlis I know it's not ideal for LSTMs, the question is more of just curiosity than anything. I've included my code in the edit $\endgroup$
    – Recessive
    Jan 18, 2020 at 2:37
  • $\begingroup$ Do you apply weight regularization? That would make difficult to output high values, even more if input values are low. $\endgroup$
    – David
    Jan 18, 2020 at 12:21

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .