Questions tagged [vanishing-gradient-problem]

For questions related to the vanishing gradient problem, which is a numerical problem that occurs while training a (deep) neural network with a gradient-based optimization technique. There's also the related exploding gradient problem.

Filter by
Sorted by
Tagged with
2
votes
2answers
54 views

What are the common pitfalls that we could face when training neural networks?

Apart from the vanishing or exploding gradient problems, what are other problems or pitfalls that we could face when training neural networks?
1
vote
1answer
43 views

Does the paper “On the difficulty of training Recurrent Neural Networks” (2013) assume, falsely, that spectral radii are $\ge$ square matrix norms?

(link to paper in arxiv) In section 2.1 the authors define $\gamma$ as the maximum possible value of the derivative of the activation function (e.g. 1 for tanh.) Then they have this to say: We ...
1
vote
0answers
11 views

How do LSTM and GRU avoid to overcome the vanishing gradient problem?

I'm watching the video Recurrent Neural Networks (RNN) | RNN LSTM | Deep Learning Tutorial | Tensorflow Tutorial | Edureka where the author says that the LSTM and GRU architecture help to reduce the ...
4
votes
1answer
53 views

What effect does batch norm have on the gradient?

Batch norm is a technique where they essentially standardize the activations at each layer, before passing it on to the next layer. Naturally, this will affect the gradient through the network. I have ...
0
votes
1answer
60 views

If vanishing gradients are NOT the problem that ResNets solve, then what is the explanation behind ResNet success?

I often see blog posts or questions on here starting with the premise that ResNets solve the vanishing gradient problem. The original 2015 paper contains the following passage in section 4.1: We ...
1
vote
0answers
31 views

Which activation functions can lead to the vanishing gradient problem?

From this video tutorial Vanishing Gradient Tutorial, the sigmoid function and the hyperbolic tangent can produce the vanishing gradient problem. What other activation functions can lead to the ...
5
votes
1answer
462 views

Why do ResNets avoid the vanishing gradient problem?

I read that, if we use the sigmoid or hyperbolic tangent activation functions in deep neural networks, we can have some problems with the vanishing of the gradient, and this is visible by the shapes ...