People who code: we want your input. Take the Survey

Questions tagged [vanishing-gradient-problem]

For questions related to the vanishing gradient problem, which is a numerical problem that occurs while training a (deep) neural network with a gradient-based optimization technique. There's also the related exploding gradient problem.

Filter by
Sorted by
Tagged with
6
votes
1answer
3k views

Why do ResNets avoid the vanishing gradient problem?

I read that, if we use the sigmoid or hyperbolic tangent activation functions in deep neural networks, we can have some problems with the vanishing of the gradient, and this is visible by the shapes ...
4
votes
1answer
860 views

How to detect vanishing gradients?

Can vanishing gradients be detected by the change in distribution (or lack thereof) of my convolution's kernel weights throughout the training epochs? And if so how? For example, if only 25% of my ...
5
votes
1answer
222 views

What effect does batch norm have on the gradient?

Batch norm is a technique where they essentially standardize the activations at each layer, before passing it on to the next layer. Naturally, this will affect the gradient through the network. I have ...
2
votes
0answers
62 views

How to decide if gradients are vanishing?

I am trying to debug a convolutional neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing ...
2
votes
1answer
191 views

If vanishing gradients are NOT the problem that ResNets solve, then what is the explanation behind ResNet success?

I often see blog posts or questions on here starting with the premise that ResNets solve the vanishing gradient problem. The original 2015 paper contains the following passage in section 4.1: We ...