When training on large neural network, how to deal with the case that the gradients are too small to have any impact?
FYI, I have an RNN, which has multiple LSTM cells and each cell has hundreds of neurons. Each training data has thousands of steps, so the RNN would unroll thousands of times. When I print out all gradients, they are very small, like e-20 of the variable values. Therefore the training does not change the variable values at all.
BTW, I think this is not an issue of vanishing gradients. Note that the gradients are uniformly small from the beginning to the end.
Any suggestion to overcome this issue?