# Is this LSTM layer learning anything?

I've trained a CNN-LSTM model but the results weren't satisfactory, so I took a look at my weight distributions and this is what I got:

I don't understand. Is this layer learning anything? Or no?

Update: I've also tried LeakyReLU activation and also removed l2 regularization and this is what I got: So I guess my layer isn't learning or does take more epochs to train LSTM layers? The gradients are not vanishing because the CNN layer before this is changing.

What may be more informative in terms of whether it is learning or not, is to track gradients.

Through gradients you will be able to understand better whether activations are receiving error terms to adjust weights accordingly or not. In the latter case, this would be characteristic of vanishing gradients problem.

You are developing with tf.keras, in which case you can add to your tensorboard callback: tf.keras.callbacks.TensorBoard(write_grads=True)

• try replacing tanh with alternative activation functions in the LSTM layers of your model tf.keras.LSTM(activation=tf.keras.layers.LeakyReLu()) (see tanh saddle points problem)
• It seems that tf.keras.callbacks.TensorBoard(write_grads=True) is deprecated in tensorflow 2.4 but I found this article on how to do it manually. Also I'll try the leaky relu too, thanks. – Sepehr Golestanian Mar 13 at 10:16