0
$\begingroup$

I'm currently working on implementing a neural network using the sigmoid activation function and the binary cross-entropy cost function. In my implementation, I've noticed that the gradient calculation for the last layer activation differs from other layers. I'm seeking a clear explanation or proof for this discrepancy.

Specifically, I'm curious about why the gradient calculation for the last layer is different and whether it's influenced by the choice of cost function and activation function. Additionally, I would appreciate insights into why we can't simply use the formula dAL = d(cost) / d(AL) = d(cost) / d(ZL) * d(ZL) / d(AL) for calculating gradients in all layers. so we don't want to find da_prev using this np.dot(W.T, dZ), we compute it using a different approach. I'm eager to understand the rationale behind this choice.

Any explanations, proofs, or insights into the reasons behind these gradient calculations would be greatly appreciated. Thank you for your help and guidance!

$\endgroup$
1
  • $\begingroup$ how are they different? Can you show us? And based on what software did you implement? $\endgroup$
    – lpounng
    May 19, 2023 at 4:47

1 Answer 1

1
$\begingroup$

In neural networks, The gradient calculation for the last layer differs from the other layers due to the specific combination of the cost function and the activation function used in that layer. for a detailed explanation take a look at this example in which I am going to explain it by using the binary cross-entropy loss function and sigmoid activation function.

The binary cross-entropy cost function for a single example is:

ϕ=−[tlog(y^)+(1−t)log(1−y^)]

where t is the target output and y_hat is the predicted output 1.

The sigmoid activation function is:

sigmoid activation

which is used to transform the linear output z of the neurons into a probability between 0 and 1.

In the backpropagation process, you calculate the derivative of the cost function concerning the weights, biases, and activations in the network. This involves applying the chain rule of differentiation, which states that the derivative of a composite function is the product of the derivatives of the composed functions.

In the last layer of the network, the derivative of the cost function concerning the activation (d_cost/d_a) is different from the other layers because the cost is directly computed from the output of the last layer. In the case of binary cross-entropy and sigmoid activation, you can derive d_cost/d_a as follows:

deriavation of loss function

This is not the case for the other layers, because the cost is not directly computed from their outputs 1.

The derivative of the cost function concerning the pre-activation (d_cost/d_z) can be obtained by applying the chain rule and using the derivative of the sigmoid function, which is sigmoid(x) * (1 - sigmoid(x)):

last layer derivative

For the other layers, the gradient of the cost concerning the pre-activation is computed by propagating the gradients back from the next layer 2.

As for the question of why we can't use dA_prev = np.dot(W.T, dZ) for calculating gradients in all layers, this formula is used to propagate the gradients to the previous layer. dA_prev represents the gradient of the cost concerning the activations of the previous layer, which is computed as the dot product of the transposed weights and the gradient of the cost concerning the pre-activations of the current layer (dZ). This is part of the backpropagation algorithm and is used in all layers, not just the last layer 2.

For more information, you can check these links which I have used as references

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .