I've trained a CNN-LSTM model but the results weren't satisfactory, so I took a look at my weight distributions and this is what I got: enter image description here

I don't understand. Is this layer learning anything? Or no?

Update: I've also tried LeakyReLU activation and also removed l2 regularization and this is what I got: enter image description here So I guess my layer isn't learning or does take more epochs to train LSTM layers? The gradients are not vanishing because the CNN layer before this is changing.


What may be more informative in terms of whether it is learning or not, is to track gradients.

Through gradients you will be able to understand better whether activations are receiving error terms to adjust weights accordingly or not. In the latter case, this would be characteristic of vanishing gradients problem.

You are developing with tf.keras, in which case you can add to your tensorboard callback: tf.keras.callbacks.TensorBoard(write_grads=True)

Additional experiments may include:

  • trying shorter length sequences and compare grad flow
  • try replacing tanh with alternative activation functions in the LSTM layers of your model tf.keras.LSTM(activation=tf.keras.layers.LeakyReLu()) (see tanh saddle points problem)

enter image description here

(graph borrowed from d2l)

  • 2
    $\begingroup$ It seems that tf.keras.callbacks.TensorBoard(write_grads=True) is deprecated in tensorflow 2.4 but I found this article on how to do it manually. Also I'll try the leaky relu too, thanks. $\endgroup$ – Sepehr Golestanian Mar 13 at 10:16
  • $\begingroup$ Oh interesting article @SepehrGolestanian, wasn't aware. I have been using tbX for pytorch and it seems to still be there. Feel free to upvote/choose as solution if answer was useful. $\endgroup$ – hH1sG0n3 Mar 15 at 10:31
  • $\begingroup$ I’ve tried leaky relu too but nothing changed $\endgroup$ – Sepehr Golestanian Mar 15 at 12:50
  • 1
    $\begingroup$ It is difficult to know if you cannot. The weight distribution of a neural network that has been trained with regularisation is expected to be distributed normally. See here relevant paper. proceedings.mlr.press/v37/blundell15.pdf $\endgroup$ – hH1sG0n3 Mar 19 at 14:35
  • 1
    $\begingroup$ I didn't know that! thanks!! $\endgroup$ – Sepehr Golestanian Apr 2 at 8:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.