I'm currently using 3Blue1Brown's tutorial series on neural networks and lack extensive calculus knowledge/experience.
I'm using the following equations to calculate the gradients for weights and biases as well as the equations to find the derivative of the cost with respect to a hidden layer neuron:
The issue is, during backpropagation, the gradients keep cancelling each other out because I take an average for opposing training examples. That is, if I have two training labels being [1, 0], [0, 1], the gradients that adjust for the first label get reversed by the second label because an average for the gradients is taken. The network simply keeps outputting the average of these two and causes the network to always output [0.5, 0.5], regardless of the input.
To prevent this, I figured a softmax function would be required for the last layer instead of a sigmoid, which I used for all the layers.
However, I have no idea how to implement this. The math is difficult to understand and the notation is complicated for me.
The equations I provided above show the term: σ'(z), which is the derivative of the sigmoid function.
If I'm using softmax, how am I supposed to substitute sigmoid with it?
If I'm not mistaken, the softmax function doesn't just take one number analogous to the sigmoid, and uses all the outputs and labels.
To sum it up, the things I'd like to know and understand are:
- The equation for the neuron in every layer besides the output is: σ(w1x1 + w2x2 + ... + wnxn + b). How am I supposed to make an analogous equation with softmax for the output layer?
- After using (1) for forward propagation, how am I supposed to replace the σ'(z) term in the equations above with something analogous to softmax to calculate the partial derivative of the cost with respect to the weights, biases, and hidden layers?
