Questions tagged [exploding-gradient-problem]
For questions related to the exploding gradient problem, which is the numerical problem associated with the significant increase (or explosion) of the numbers of the gradient vector of an objective function with respect to the parameters of a neural network, which is being trained with a gradient-based optimization algorithm and backpropagation. There is also the related vanishing gradient problem, which arises when the numbers become very small.
1
question with no upvoted or accepted answers
1
vote
0
answers
22
views
Can an activation function with large derivative cause exploding gradient?
The maximum derivative of most of the currently existing activation functions is around 1. Can an activation function with derivatives higher than 1, say 1000 (a), cause exploding gradient problem? ...
- 11