Backpropagation with CrossEntropy and Softmax, HOW?

Question

Let Zs be the input of the output layer (for example, Z1 is the input of the first neuron in the output layer), Os be the output of the output layer (which are actually the results of applying the softmax activation function to Zs, for example, O1 = softmax(Z1)), and Ys be the target values (which are 0 or 1 because in this example we are dealing with classification problems and using one-hot encoding). E is the sum of the neuron's loss using the CrossEntropy loss function.

Let's say our neural network has 2 neurons, and Y1 = 1 (so Y2 = 0). What is the derivative of E with respect to Z1 and the derivative of E with respect to Z2? After calculations, I came to the conclusion that the value of all derivative of E with respects to Zs(Z1 and Z2) should be equal, becasue they are all equal to O1-1 ( since Y1 = 1 as i said), so am i right or wrong?(and why)

Answer 1

No it's not the same. First the derivative of the cost function is taken with respect to the weight and not the input. This is usually done using the chain rule of calculus. To calculate this using the chain rule we take the derivatives of the activation of each neuron(I.e the derivative of the activation function which takes in input from the previous layer and is parameterized). So the derivative of the cost function with respect to the weight is dependent on the position of the weight and the activation functions that came before it(from the previous layers). If their activations differ then their derivative isn't the same.

So the derivatives of E with respect to the weights of Z1 and Z2 are different.

Answer 2

The derivation of the softmax function is a bit tricky because except the other common activation functions(sigmoid, relu...) all the Zs values impact each other. It is because when you are calculating the value of the softmax function at the denominator you are summing all the exponentials of the layer's values. For short, one input value(which could be Z1 in this case) of the softmax function has an impact on all output values(which are Os in this case).

So, you need to calculate separately for each output value(Os) with respect to on of the input values(Z1 for example) and add them to get the derivative of E wrt(with respect to) Z1. Or in other words you need to calculate the derivative of O1 wrt Z1, then derivative of O2 wrt Z1 and when you add the two values you will get the derivative of the E wrt Z1(comes from calculus).

Now let me show you the derivatives of E wrt Z1 and Z2 won't be same.