I am trying to train a RNN with text from wikipedia but I having having trouble getting the RNN to converge. I have tried increasing the batch size but it doesn't seem to be helping. All data is one hot encoded before being used and I am using the Adam optimizer which is implemented like this.

   for k in M.keys(): ##For k in weights
        M[k] = beta1 * M[k] + (1-beta1)*grad[k]
        R[k] = beta2 *R[k] + (1-beta2)*grad[k]**2
        m_k = M[k] / (1-beta1**n)
        r_k = R[k] / (1-beta2**n)
        model[k] = model[k] - alpha * m_k / np.sqrt(r_k + 1e-8)

Beta1 is set to 0.9, beta2 to 0.999 and alpha is set to 0.001. When I train it for 50,000 I get very high fluctuation of the cost and it never seems to significantly decrease (only sometimes due to the fluctuations (and I catch the weights with the lowest cost)).My hidden_size is 400 and the batch size is 200

After sketching the cost of iterations I get a graph like this:]

It seems to be increasing on average only seeming to decrease to the the large fluctuations. What can I change to have better success and have it converge?

Thanks for any help

Edit: I plotted the norm of the gradient using the slightly different cost function which @DennisSoemers suggested and the gradient does not reduce significantly.

Sorry the title should read "Gradient over iterations"

  • $\begingroup$ Have you tried a different optimiser? I find Adadelta to be a much more robust than Adam in general. Also, in some cases you can benefit from using a simpler cell - maybe try a GRU instead and see if that helps. Finally, the batch size seems a bit high - try to reduce it by an order of magnitude. $\endgroup$
    – cantordust
    Dec 27 '18 at 13:32
  • 1
    $\begingroup$ When you train RNNs, it's crucial to use gradient clipping. Could you draw a norm of a gradient over time? $\endgroup$ Dec 28 '18 at 16:15
  • $\begingroup$ @KonstantinSolomatov I have added the graph of the norm of the gradient over time would gradient clipping be of any use? $\endgroup$
    – treutm
    Dec 30 '18 at 20:19
  • $\begingroup$ @treutm It looks like you added the same plot twice, rather than two different plots? As for cross-entropy loss, normally that would be $-y \log \hat{y}$, where $y$ is your target and $\hat{y}$ your current prediction. Your loss function definitely should somehow include both the target and your current prediction... it looks like what you wrote only contains the target, not the current prediction? $\endgroup$
    – Dennis Soemers
    Dec 30 '18 at 20:25
  • 1
    $\begingroup$ @treutm You're doing character prediction? If so ehm.. yes, I think so. I personally don't really often work with text-based ML, but... yes. $y$ and $\hat{y}$ should have the same format. If $\hat{y}$ is a vector encoding a discrete probability distribution over characters, $y$ should also be a vector of equal length encoding the ground truth (i.e. a single $1$ entry, all other entries $0$). For the math to still be correct with vector-valued outputs, we'll have to transpose the first one in what I wrote above; $-y^{\top} \log \hat{y}$ $\endgroup$
    – Dennis Soemers
    Dec 30 '18 at 20:37

Adam has combined features of RMSProp and Momentum, which sometimes perform bad on Recurrent network. In my own experience, RMSProp give good convergence rate.

Another: you can try NADAM, which is RMSProp+Nestorov-Momentum. This adds a correction step over momentum. On text-dataset, it has performed well as compared to other optimizers.

Note: Optimizers can not be only reason for convergence, it depend on many others factors like network architecture, activations, input data-set, encoding and so on.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.