Difference in gradient calculation for the last layer activation in neural networks

Question

I'm currently working on implementing a neural network using the sigmoid activation function and the binary cross-entropy cost function. In my implementation, I've noticed that the gradient calculation for the last layer activation differs from other layers. I'm seeking a clear explanation or proof for this discrepancy.

Specifically, I'm curious about why the gradient calculation for the last layer is different and whether it's influenced by the choice of cost function and activation function. Additionally, I would appreciate insights into why we can't simply use the formula dAL = d(cost) / d(AL) = d(cost) / d(ZL) * d(ZL) / d(AL) for calculating gradients in all layers. so we don't want to find da_prev using this np.dot(W.T, dZ), we compute it using a different approach. I'm eager to understand the rationale behind this choice.

Any explanations, proofs, or insights into the reasons behind these gradient calculations would be greatly appreciated. Thank you for your help and guidance!

Answer 1

In neural networks, The gradient calculation for the last layer differs from the other layers due to the specific combination of the cost function and the activation function used in that layer. for a detailed explanation take a look at this example in which I am going to explain it by using the binary cross-entropy loss function and sigmoid activation function.

The binary cross-entropy cost function for a single example is:

ϕ=−[tlog(y^)+(1−t)log(1−y^)]

where t is the target output and y_hat is the predicted output 1.

The sigmoid activation function is:

which is used to transform the linear output z of the neurons into a probability between 0 and 1.

In the backpropagation process, you calculate the derivative of the cost function concerning the weights, biases, and activations in the network. This involves applying the chain rule of differentiation, which states that the derivative of a composite function is the product of the derivatives of the composed functions.

In the last layer of the network, the derivative of the cost function concerning the activation (d_cost/d_a) is different from the other layers because the cost is directly computed from the output of the last layer. In the case of binary cross-entropy and sigmoid activation, you can derive d_cost/d_a as follows:

This is not the case for the other layers, because the cost is not directly computed from their outputs 1.

The derivative of the cost function concerning the pre-activation (d_cost/d_z) can be obtained by applying the chain rule and using the derivative of the sigmoid function, which is sigmoid(x) * (1 - sigmoid(x)):

For the other layers, the gradient of the cost concerning the pre-activation is computed by propagating the gradients back from the next layer 2.

As for the question of why we can't use dA_prev = np.dot(W.T, dZ) for calculating gradients in all layers, this formula is used to propagate the gradients to the previous layer. dA_prev represents the gradient of the cost concerning the activations of the previous layer, which is computed as the dot product of the transposed weights and the gradient of the cost concerning the pre-activations of the current layer (dZ). This is part of the backpropagation algorithm and is used in all layers, not just the last layer 2.

For more information, you can check these links which I have used as references

• Question about Gradient of Cost Functions with Respect to Activations of Last Layer of Network (aka the predictions y^ )

• Meduim Article on sigmoid activation and binary cross entropy