What may be more informative in terms of whether it is learning or not, is to track gradients.
Through gradients you will be able to understand better whether activations are receiving error terms to adjust weights accordingly or not. In the latter case, this would be characteristic of vanishing gradients problem.
You are developing with tf.keras, in which case you can add to your tensorboard callback:
Additional experiments may include:
- trying shorter length sequences and compare grad flow
- try replacing
tanh with alternative activation functions in the LSTM layers of your model
tanh saddle points problem)
(graph borrowed from d2l)