Numerical problems with gradient descent

Question

I'm trying to implement a simple neural network for classification (multi-class) as an exercise (written in C). During gradient descent, the weights and biases quickly get out of control and the gradient becomes infinite.

I haven't been able to find any discussion of such problems (vanishing gradients is kind of the opposite).

To be more specific, for testing I use a very simple network with 1 hidden layer and sigmoid as activation function. For the output layer I use softmax and logarithmic loss.

The issue as I see it is that when an output activation is very small, for the derivative of the loss I basically get 1 / <very_small number> and this leads to an enormous gradient.

Am I doing something wrong in terms of network architecture? What is the typical way to deal with such problems?

Answer 1

You're describing exploding gradients, which as you noted is the opposite issue of vanishing gradients. Both these problems arise from the process of propagating and backpropagating activations/errors through the network. The intuition is that repeated multiplications of numbers less than 1 rapidly leads to a product of 0, while repeated multiplications of numbers greater than 1 rapidly leads to a product which tends to infinity.

The number of matrix multiplications you need to perform increases with the number of layers, which is why this tends to be a problem with deeper networks. In particular, recurrent neural networks are infamously susceptible to vanishing/exploding gradients as a result of propagating through each layer multiple times corresponding to each time step.

Careful choice of activation functions, proper weight initialization, and gradient clipping are the standard ways to address this problem, though more specialized solutions can arise depending on the context (for example, for RNNs, entirely new types of neurons, LSTMs and GRUs, were developed to address the issue).