# RNN LSTM not converging with Adam

I am trying to train a RNN with text from wikipedia but I having having trouble getting the RNN to converge. I have tried increasing the batch size but it doesn't seem to be helping. All data is one hot encoded before being used and I am using the Adam optimizer which is implemented like this.

   for k in M.keys(): ##For k in weights
M[k] = beta1 * M[k] + (1-beta1)*grad[k]
R[k] = beta2 *R[k] + (1-beta2)*grad[k]**2
m_k = M[k] / (1-beta1**n)
r_k = R[k] / (1-beta2**n)
model[k] = model[k] - alpha * m_k / np.sqrt(r_k + 1e-8)


Beta1 is set to 0.9, beta2 to 0.999 and alpha is set to 0.001. When I train it for 50,000 I get very high fluctuation of the cost and it never seems to significantly decrease (only sometimes due to the fluctuations (and I catch the weights with the lowest cost)).My hidden_size is 400 and the batch size is 200

After sketching the cost of iterations I get a graph like this:

It seems to be increasing on average only seeming to decrease to the the large fluctuations. What can I change to have better success and have it converge?

Thanks for any help

Edit: I plotted the norm of the gradient using the slightly different cost function which @DennisSoemers suggested and the gradient does not reduce significantly.

• Have you tried a different optimiser? I find Adadelta to be a much more robust than Adam in general. Also, in some cases you can benefit from using a simpler cell - maybe try a GRU instead and see if that helps. Finally, the batch size seems a bit high - try to reduce it by an order of magnitude. – cantordust Dec 27 '18 at 13:32
• When you train RNNs, it's crucial to use gradient clipping. Could you draw a norm of a gradient over time? – Konstantin Solomatov Dec 28 '18 at 16:15
• @KonstantinSolomatov I have added the graph of the norm of the gradient over time would gradient clipping be of any use? – treutm Dec 30 '18 at 20:19
• @treutm It looks like you added the same plot twice, rather than two different plots? As for cross-entropy loss, normally that would be $-y \log \hat{y}$, where $y$ is your target and $\hat{y}$ your current prediction. Your loss function definitely should somehow include both the target and your current prediction... it looks like what you wrote only contains the target, not the current prediction? – Dennis Soemers Dec 30 '18 at 20:25
• @treutm You're doing character prediction? If so ehm.. yes, I think so. I personally don't really often work with text-based ML, but... yes. $y$ and $\hat{y}$ should have the same format. If $\hat{y}$ is a vector encoding a discrete probability distribution over characters, $y$ should also be a vector of equal length encoding the ground truth (i.e. a single $1$ entry, all other entries $0$). For the math to still be correct with vector-valued outputs, we'll have to transpose the first one in what I wrote above; $-y^{\top} \log \hat{y}$ – Dennis Soemers Dec 30 '18 at 20:37