Edit: I've reworked my question to generalize better and be more on-topic, and be mostly software implementation agnostic.

Can vanishing gradients be detected by the change in distribution (or lack thereof) of my convolution's kernel weights throughout the training epochs? And if so how?

For example, if only 25% of my kernel's weights ever change throughout the epochs, does that imply an issue with vanishing gradients?

Here are my histograms and distributions, is it possible to tell whether my model suffers from Vanishing gradients from these images? (some middle hidden layers omitted for brevity) Thanks in advance.

enter image description here enter image description here enter image description here enter image description here

  • $\begingroup$ Hi and welcome to AI SE! Please, simplify your post and ask only one question per post. Please, ask the other questions in their separate post. Maybe some of us know the answer to one question but not the others. $\endgroup$ – nbro Feb 22 at 2:54
  • $\begingroup$ @nbro Hi nbro, thanks. Will do 👍 $\endgroup$ – Elegant Code Feb 24 at 18:43

Vanishing Gradients can be detected from the kernel weights distribution. All you have to look for is whether the weights are dying down to 0.

If only 25% of your kernel weights are changing that does not imply vanishing gradient, it might be a factor but there can be a variety of reasons. Starting from poor data, loss function used to the optimizer, etc. Kernel weight not changing only points out that the model is not learning well.

From the histograms, only the conv_2d_2 layer shows any form of vanishing gradients since the numbers are pretty small. But even that seems to be picking after 600 epochs. Usually, a good indicator is to have the mean of weights in a layer to be closer to 0 and Standard Deviation to be closer to 1. So if this is maintained to a good extent during training your good to go.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.