I'm trying to implement a simple neural network for classification (multi-class) as an exercise (written in C). During gradient descent, the weights and biases quickly get out of control and the gradient becomes infinite.
I haven't been able to find any discussion of such problems (vanishing gradients is kind of the opposite).
To be more specific, for testing I use a very simple network with 1 hidden layer and sigmoid as activation function. For the output layer I use softmax and logarithmic loss.
The issue as I see it is that when an output activation is very small, for the derivative of the loss I basically get
1 / <very_small number> and this leads to an enormous gradient.
Am I doing something wrong in terms of network architecture? What is the typical way to deal with such problems?