My cross entropy loss gradient calculation is wrong according to the answer key

Given a neural network model for Covid-19 classification with $$C=1$$ for positive and $$C=0$$ for negative Let $$x_1 = 6$$ and $$x_2=2$$ find

1. Probability if the patient got Covid-19 $$p\left(C=1 | x; w,b\right)$$
2. Probability if the patient didn’t get Covid-19 $$p\left(C=0 | x; w,b\right)$$
3. Find the gradient of $$CE_{Loss}$$

My attempt

For the first problem \begin{aligned} O_1 &= \text{ReLU} \left( b_1 + \sum{x_iw_i} \right) \\ &= \text{ReLU} \left( 0.2 + 6 \cdot 0.3 - 0.2 \cdot 2 \right) \\ &= \text{ReLU} \left( 1.6 \right) \\ &= 1.6 \\ \end{aligned}

\begin{aligned} O_2 &= \text{Sig} \left( -0.6 - 0.1 \cdot 2 - 0.2 \cdot 6 \right) \\ &= \text{Sig} \left( -2 \right) \\ &\approx 0.8808 \\ \end{aligned}

\begin{aligned} O_3 &= \text{Sig} \left( 0.6 - 0.3 \cdot 0.8808 + 1.6 \cdot 0.5 \right) \\ &\approx 0.75690 \\ \end{aligned}

\begin{aligned} p\left(C=1 | x; w,b\right) = O_3 \approx 0.75690 \end{aligned}

For the second problem \begin{aligned} p\left(C=0 | x; w,b\right) = 1 - \left(C=1 | x; w,b\right) = 0.2431 \end{aligned}

For the third problem

Since it is a lot of things to calculate I'll take $$\frac{\delta L}{\delta w_6}$$ as example \begin{aligned} \frac{\delta L}{\delta w_6} &= \frac{\delta L}{\delta \hat{y}} \cdot \frac{\delta \hat{y}}{\delta w_6} \\ &= \left(\hat{y}-y\right)O_2\left(O_3 \left(1-O_3\right)\right)\\ &= \left(O_3-y\right)O_2\left(O_3 \left(1-O_3\right)\right) \end{aligned} The answer key says it should be $$\left(O_3-y\right)O_2$$. Where did I go wrong?

• Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
– Community Bot
Dec 4, 2022 at 17:27

For the first problem your $$O_2$$ is mistaken and should be corrected as below \begin{aligned} O_2 &= \text{Sig} \left( -0.6 - 0.1 \cdot 2 + 0.2 \cdot 6 \right) \\ &= \text{Sig} \left( 0.4 \right) \\ &\approx 0.5987 \\ \end{aligned}

\begin{aligned} O_3 &= \text{Sig} \left( 0.6 - 0.3 \cdot 0.5987 + 1.6 \cdot 0.5 \right) \\ &\approx 0.7721 \\ \end{aligned}

\begin{aligned} p\left(C=1 | x; w,b\right) = O_3 \approx 0.7721 \end{aligned}

For the second problem thus corrected to: \begin{aligned} p\left(C=0 | x; w,b\right) = 1 - \left(C=1 | x; w,b\right) = 0.2279 \end{aligned}

For the third problem since $$w_6$$ is one of the weights connecting the output layer and we assume $$\hat{y}=g(z^{(3)})$$, where $$z^{(3)}$$ is the net input for the output unit and $$g$$ is its sigmoid transfer function. Then we arrive at: \begin{aligned} \frac{\delta L}{\delta w_6} &= \frac{\delta L}{\delta \hat{y}} \cdot \frac{\delta \hat{y}}{\delta z^{(3)}} \cdot \frac{\delta z^{(3)}}{\delta w_6} \\ \end{aligned}

and since $$L$$ is the binary cross-entropy loss function, we know $$\frac{\delta L}{\delta \hat{y}} = \frac{(\hat{y}-y)}{(1-\hat{y})\hat{y}}$$. And from above transfer function we know $$\frac{\delta \hat{y}}{\delta z^{(3)}}=(1-\hat{y})\hat{y}$$, and $$\frac{\delta z^{(3)}}{\delta w_6}=O_2$$. Thus finally we simplify as $$\frac{\delta L}{\delta w_6}=(O_3-y)O_2$$

• Thanks for your response. One more thing, for $\frac{\delta L}{\delta w_4}$ is it $\frac{\delta L}{\delta \hat{y}} \cdot \frac{\delta \hat{y}}{\delta z^{(3)}} \cdot \frac{\delta z^{(3)}}{\delta z^{(2)}} \cdot \frac{ \delta z^{(2)}}{\delta w_4} = (O_3-y)(1-O_2)O_2x_2$? Dec 5, 2022 at 15:30
• For the hidden layer it's more complex and you need to further chain $\frac{\delta z^{(3)}}{\delta z^{(2)}}$ into $\frac{\delta z^{(3)}}{\delta O_2} \cdot \frac{\delta O_2}{\delta z^{(2)}}$, and then try to simplify from there. Dec 6, 2022 at 0:55