3
$\begingroup$

When training on large neural network, how to deal with the case that the gradients are too small to have any impact?

FYI, I have an RNN, which has multiple LSTM cells and each cell has hundreds of neurons. Each training data has thousands of steps, so the RNN would unroll thousands of times. When I print out all gradients, they are very small, like e-20 of the variable values. Therefore the training does not change the variable values at all.

BTW, I think this is not an issue of vanishing gradients. Note that the gradients are uniformly small from the beginning to the end.

Any suggestion to overcome this issue?

Thank!

Improve this question
$\endgroup$
3
  • $\begingroup$ Welcome to ai.se...great question, low gradient means your error is also low...why do u need a high accuracy? Also you can increase the floating point representation if you want high accuracy, or maybe increase the learning rate $\endgroup$
    – user9947
    Commented Apr 6, 2018 at 6:55
  • $\begingroup$ Did you tried Backprogation Through Time (BPTT) ? Need more information about your net and task to help you $\endgroup$
    – user3352632
    Commented Apr 9, 2018 at 16:28
  • $\begingroup$ Thanks for your comment and suggestion. I changed the layer from tf.contrib.rnn.LSTMBlockCell to tf.contrib.rnn.LayerNormBasicLSTMCell. Then the gradients become large enough to influence the network. $\endgroup$
    – Tom Z
    Commented Apr 11, 2018 at 5:29

2 Answers 2

Reset to default
3
$\begingroup$

Vanishing gradient is a common problem in RNN.

A common way to deal with it is the method of gradient clipping (mainly you define a maximum and/ or a minimum threshold). see here for more information

Further information and piece of code to implement it can be found in SO here

Hope it helps !

Improve this answer
$\endgroup$
2
$\begingroup$

I changed the layer from tf.contrib.rnn.LSTMBlockCell to tf.contrib.rnn.LayerNormBasicLSTMCell. Then the gradients become large enough to influence the network.

Improve this answer
$\endgroup$
2
  • $\begingroup$ You need to add details about what exactly does that mean for those who don't know tensorflow $\endgroup$
    – user9947
    Commented Apr 11, 2018 at 5:50
  • $\begingroup$ It means that I normalize the input for the network layers. I hope this explains. $\endgroup$
    – Tom Z
    Commented Apr 13, 2018 at 17:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .