# RNN LSTM not converging with Adam

I am trying to train a RNN with text from wikipedia but I having having trouble getting the RNN to converge. I have tried increasing the batch size but it doesn't seem to be helping. All data is one hot encoded before being used and I am using the Adam optimizer which is implemented like this.

   for k in M.keys(): ##For k in weights
M[k] = beta1 * M[k] + (1-beta1)*grad[k]
R[k] = beta2 *R[k] + (1-beta2)*grad[k]**2
m_k = M[k] / (1-beta1**n)
r_k = R[k] / (1-beta2**n)
model[k] = model[k] - alpha * m_k / np.sqrt(r_k + 1e-8)


Beta1 is set to 0.9, beta2 to 0.999 and alpha is set to 0.001. When I train it for 50,000 I get very high fluctuation of the cost and it never seems to significantly decrease (only sometimes due to the fluctuations (and I catch the weights with the lowest cost)).My hidden_size is 400 and the batch size is 200

After sketching the cost of iterations I get a graph like this:

It seems to be increasing on average only seeming to decrease to the the large fluctuations. What can I change to have better success and have it converge?

Thanks for any help

Edit: I plotted the norm of the gradient using the slightly different cost function which @DennisSoemers suggested and the gradient does not reduce significantly.

Sorry the title should read "Gradient over iterations"

• Have you tried a different optimiser? I find Adadelta to be a much more robust than Adam in general. Also, in some cases you can benefit from using a simpler cell - maybe try a GRU instead and see if that helps. Finally, the batch size seems a bit high - try to reduce it by an order of magnitude. – cantordust Dec 27 '18 at 13:32
• When you train RNNs, it's crucial to use gradient clipping. Could you draw a norm of a gradient over time? – Konstantin Solomatov Dec 28 '18 at 16:15
• @KonstantinSolomatov I have added the graph of the norm of the gradient over time would gradient clipping be of any use? – treutm Dec 30 '18 at 20:19
• @treutm It looks like you added the same plot twice, rather than two different plots? As for cross-entropy loss, normally that would be $-y \log \hat{y}$, where $y$ is your target and $\hat{y}$ your current prediction. Your loss function definitely should somehow include both the target and your current prediction... it looks like what you wrote only contains the target, not the current prediction? – Dennis Soemers Dec 30 '18 at 20:25
• @DennisSoemers Whoops I just realized that too :). When I use one-hot encoding would y = 0 for all other characters in the vector so they don't have an effect? – treutm Dec 30 '18 at 20:27

## 1 Answer

Adam has combined features of RMSProp and Momentum, which sometimes perform bad on Recurrent network. In my own experience, RMSProp give good convergence rate.

Another: you can try NADAM, which is RMSProp+Nestorov-Momentum. This adds a correction step over momentum. On text-dataset, it has performed well as compared to other optimizers.

Note: Optimizers can not be only reason for convergence, it depend on many others factors like network architecture, activations, input data-set, encoding and so on.