2
$\begingroup$

I am trying to debug a convolutional neural network. I am seeing gradients close to zero.

How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing gradient by looking at the values?

I am getting values close to $4$ decimal places (e.g. $0.0001$) and, in some cases, close to $5$ decimal places (e.g. $0.00001$).

enter image description here

The CNN seems not to be learning since the histogram of weight is also quite similar in all epochs.

I am using the ReLU activation function and Adam optimizer. What could be the reason for the vanishing gradient in the case of the ReLU activation function?

If it is possible, please, point me to some resources that might be helpful.

Improve this question
$\endgroup$
1
  • $\begingroup$ This question is partially a duplicate of this one, but here you're also asking about vanishing gradients in the context of ReLU, so I will not mark it as a duplicate. $\endgroup$
    – nbro
    Commented Dec 13, 2020 at 13:20

0

Reset to default

You must log in to answer this question.