Skip to main content

Questions tagged [exploding-gradient-problem]

For questions related to the exploding gradient problem, which is the numerical problem associated with the significant increase (or explosion) of the numbers of the gradient vector of an objective function with respect to the parameters of a neural network, which is being trained with a gradient-based optimization algorithm and backpropagation. There is also the related vanishing gradient problem, which arises when the numbers become very small.

Filter by
Sorted by
Tagged with
0 votes
0 answers

Covering missing derivations in Bengio RNN exploding gradient/output paper

While I have no doubts about the soundness of its conclusions given that it is very well-known, I have been looking through the classic paper on RNNs which is used to substantiate the claims that ...
Chris's user avatar
  • 101
2 votes
1 answer

What is the justification for this approach of clipping elementwise?

I'm new to the field of AI (though I have a background in mathematics). As I was going through some documents, I read that there is a form of gradient clipping where the elements of the gradient that ...
Ukn0wn's user avatar
  • 21
1 vote
0 answers

Can an activation function with large derivative cause exploding gradient?

The maximum derivative of most of the currently existing activation functions is around 1. Can an activation function with derivatives higher than 1, say 1000 (a), cause exploding gradient problem? ...
JGM's user avatar
  • 11
1 vote
1 answer

Mathematically speaking, Is it only the product operation used in the chain rule causing the vanishing or exploding gradient?

I am asking this question from the mathematical perspective of the vanishing and exploding gradient problems that we face generally during training deep neural networks. The chain rule of ...
hanugm's user avatar
  • 3,930
5 votes
2 answers

What are the common pitfalls that we could face when training neural networks?

Apart from the vanishing or exploding gradient problems, what are other problems or pitfalls that we could face when training neural networks?
pjoter's user avatar
  • 51
2 votes
1 answer

Which work originally introduced gradient clipping?

The Deep Learning book mentions that it's been used for years but the oldest sources it mentions are from 2012: A simple type of solution has been in use by practitioners for many years: clipping ...
Jeremiah England's user avatar
5 votes
1 answer

What effect does batch norm have on the gradient?

Batch norm is a technique where they essentially standardize the activations at each layer, before passing it on to the next layer. Naturally, this will affect the gradient through the network. I have ...
information_interchange's user avatar
7 votes
1 answer

Why do ResNets avoid the vanishing gradient problem?

I read that, if we use the sigmoid or hyperbolic tangent activation functions in deep neural networks, we can have some problems with the vanishing of the gradient, and this is visible by the shapes ...
FraMan's user avatar
  • 199
8 votes
2 answers

How to deal with large (or NaN) neural network's weights?

My weights go from being between 0 and 1 at initialization to exploding into the tens of thousands in the next iteration. In the 3rd iteration, they become so large that only arrays of nan values are ...
FeedMeInformation's user avatar