I am trying to debug a convolutional neural network. I am seeing gradients close to zero.

How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing gradient by looking at the values?

I am getting values close to $4$ decimal places (e.g. $0.0001$) and, in some cases, close to $5$ decimal places (e.g. $0.00001$).

enter image description here

The CNN seems not to be learning since the histogram of weight is also quite similar in all epochs.

I am using the ReLU activation function and Adam optimizer. What could be the reason for the vanishing gradient in the case of the ReLU activation function?

If it is possible, please, point me to some resources that might be helpful.

  • $\begingroup$ This question is partially a duplicate of this one, but here you're also asking about vanishing gradients in the context of ReLU, so I will not mark it as a duplicate. $\endgroup$ – nbro Dec 13 '20 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.