# Numerical problems with gradient descent

I'm trying to implement a simple neural network for classification (multi-class) as an exercise (written in C). During gradient descent, the weights and biases quickly get out of control and the gradient becomes infinite.

I haven't been able to find any discussion of such problems (vanishing gradients is kind of the opposite).

To be more specific, for testing I use a very simple network with 1 hidden layer and sigmoid as activation function. For the output layer I use softmax and logarithmic loss.

The issue as I see it is that when an output activation is very small, for the derivative of the loss I basically get 1 / <very_small number> and this leads to an enormous gradient.

Am I doing something wrong in terms of network architecture? What is the typical way to deal with such problems?

• have you tried to search "exploding gradients"? It's a well known issue along with vanishing gradients. Most of the time both problems comes from the weight initialization, what strategy are you using for that? Aug 1, 2022 at 12:34
• @EdoardoGuerriero I'm using random values in [-1, 1]. I didn't know the term "exploding gradients", I'll check what info I can find, thanks. Aug 1, 2022 at 14:08
• highly chances that your problem comes from the initialization. Usually the weights are positive defined, and you should go for values with mean of lower magnitude, like 0.05. Uniform or normal distributions are fine, but check also xavier and kaiming initialization. Aug 1, 2022 at 15:00

## 1 Answer

You're describing exploding gradients, which as you noted is the opposite issue of vanishing gradients. Both these problems arise from the process of propagating and backpropagating activations/errors through the network. The intuition is that repeated multiplications of numbers less than 1 rapidly leads to a product of 0, while repeated multiplications of numbers greater than 1 rapidly leads to a product which tends to infinity.

The number of matrix multiplications you need to perform increases with the number of layers, which is why this tends to be a problem with deeper networks. In particular, recurrent neural networks are infamously susceptible to vanishing/exploding gradients as a result of propagating through each layer multiple times corresponding to each time step.

Careful choice of activation functions, proper weight initialization, and gradient clipping are the standard ways to address this problem, though more specialized solutions can arise depending on the context (for example, for RNNs, entirely new types of neurons, LSTMs and GRUs, were developed to address the issue).