# In gradient descent's update rule, why do we use $\sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}$ instead of $\frac{\delta C_0}{\delta w^{l}}$?

I am trying to code a two layered neural network simple NN as I have described here https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/

I am getting stuck on the last step of updating the weights after calculating the gradients for the outer and inner layers via back-propagation

#---------------------------------------------------------------

# Two layered NW. Using from (1) and the equations we derived as explanations
#---------------------------------------------------------------

import numpy as np
# seed random numbers to make calculation deterministic
np.random.seed(1)

# pretty print numpy array
np.set_printoptions(formatter={'float': '{: 0.3f}'.format})

# let us code our sigmoid funciton
def sigmoid(x):
return 1/(1+np.exp(-x))

# let us add a method that takes the derivative of x as well
def derv_sigmoid(x):
return x*(1-x)

# set learning rate as 1 for this toy example
learningRate =  1

# input x, also used as the training set here
x = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1]  ])

# desired output for each of the training set above
y = np.array([[0,1,1,0]]).T

# Explanaiton - as long as input has two ones, but not three, ouput is One
"""
Input [0,0,1]  Output = 0
Input [0,1,1]  Output = 1
Input [1,0,1]  Output = 1
Input [1,1,1]  Output = 0
"""

input_rows = 4
# Randomly initalised weights
weight1 =  np.random.random((3,input_rows))
weight2 =  np.random.random((input_rows,1))

print("Shape weight1",np.shape(weight1)) #debug
print("Shape weight2",np.shape(weight2)) #debug

# Activation to layer 0 is taken as input x
a0 = x

iterations = 1000
for iter in range(0,iterations):

# Forward pass - Straight Forward
z1= x @ weight1
a1 = sigmoid(z1)
z2= a1 @ weight2
a2 = sigmoid(z2)

# Backward Pass - Backpropagation
delta2  = (y-a2)
#---------------------------------------------------------------
# Calcluating change of Cost/Loss wrto weight of 2nd/last layer
# Eq (A) ---> dC_dw2 = delta2*derv_sigmoid(z2)
#---------------------------------------------------------------

dC_dw2  = delta2 * derv_sigmoid(a2)

if iter == 0:
print("Shape dC_dw2",np.shape(dC_dw2)) #debug

#---------------------------------------------------------------
# Calcluating change of Cost/Loss wrto weight of 2nd/last layer
# Eq (B)---> dC_dw1 = derv_sigmoid(a1)*delta2*derv_sigmoid(a2)*weight2
# note  delta2*derv_sigmoid(a2) == dC_dw2
# dC_dw1 = derv_sigmoid(a1)*dC_dw2*weight2
#---------------------------------------------------------------

dC_dw1 =  (np.multiply(dC_dw2,weight2.T)) * derv_sigmoid(a1)
if iter == 0:
print("Shape dC_dw1",np.shape(dC_dw1)) #debug

#---------------------------------------------------------------
#---------------------------------------------------------------

#weight2 = weight2 - learningRate*dC_dw2 --> these are what the textbook tells
#weight1 = weight1 - learningRate*dC_dw1

weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2) # this is what works
weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1)

print("New ouput\n",a2)



Why is

  weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2)
weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1)


  #weight2 = weight2 - learningRate*dC_dw2
#weight1 = weight1 - learningRate*dC_dw1


I am not getting the source of the equation of updating the weights by multiplying with the activation of the previous layer

As per gradient descent, the weight update should be

$$W^{l}_{new} = W^{l}_{old} - \gamma * \frac{\delta C_0}{\delta w^{l}}$$

However, what works in practice is

$$W^{l}_{new} = W^{l}_{old} - \gamma * \sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}},$$

where $$\gamma$$ is the learning rate.

• Have you read something about back-propagation before asking this question? It seems that you're asking us how is back-propagation done because you're asking us to explain a certain implementation.
– nbro
Mar 19 at 12:10
• sorry if this is a bit silly; yes - I have derived it and trying to code through (this is what I have done so far itisexplained.com/html/NN/ml/5_codingneuralnetwork) Wl−1new=Wl−1old−learningRate∗δC0/δwl−1 - is what I have understood the gradeint descent part Mar 19 at 12:12
• So, is your question: why do they use np.dot(a0.T,dC_dw1) rather than dC_dw1  in the update rule? Maybe you can explain these symbols and maybe the code is not strictly necessary and you can write the equations that they are using in the implementation with latex/mathjax.
– nbro
Mar 19 at 12:14
• It may also be a good idea to describe the neural network, the loss function, the task you're solving, etc. Back-propagation can change depending on these parameters.
– nbro
Mar 19 at 12:16
• Okay - I guess this is the matrix calculus part - explained.ai/matrix-calculus/#sec6.2 need to check this more Mar 20 at 19:06

Okay - the answer is here https://explained.ai/matrix-calculus/#sec6.2 and it is pretty involved. Basically, there is a difference when you derive the equation for one neuron and when you have to do practically for a set of neurons. The answer is matrix calculus. Here goes from what I could make out. Feel free to correct if I am wrong

## Gradient Vector/Matrix/2D tensor of Loss function wrto Weight

$$C = \frac{1}{2} \sum_j (y_j-a^L_j)^2$$

Assuming a neural net with 2 layers, we have the final Loss as

$$C = \frac{1}{2} \sum_j (y_j-a^2_j)^2$$

Where

$$a^2 = \sigma(w^2.a^1)$$

We can then write

$$C = \frac{1}{2} \sum_j v^2 \quad \rightarrow (Eq \;A)$$

Where

$$v= y-a^2$$

## Partial Derivative of Loss function wrto Weight

For the last layer, lets use Chain Rule to split like below

$$\frac {\partial C}{\partial w^2} = \frac{\partial v^2}{\partial v} * \frac{\partial v}{\partial w^2} \quad \rightarrow (Eq \;B)$$

$$\frac{\partial v^2}{\partial v} =2v \quad \rightarrow (Eq \;B.1)$$

$$\frac{\partial v}{\partial w^2}= \frac{\partial (y-a^2)}{\partial w^2} = 0-\frac{\partial a^2}{\partial w^2} \quad \rightarrow (Eq \;B.2)$$

$$\frac {\partial C}{\partial w^2} = \frac{1}{2} *2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)$$

### Now we need to find $$\frac{\partial a^2}{\partial w^2}$$

Let

$$a^2= \sigma(sum(w^2 \otimes a^1 )) = \sigma(z^2)$$ $$z^2 = sum(w^2 \otimes a^1 )$$

$$z^2 = sum(k^2) \; \text {where} \; k^2=w^2 \otimes a^1$$

We now need to derive an intermediate term which we will use later

$$\frac{\partial z^2}{\partial w^2} =\frac{\partial z^2}{\partial k^2}*\frac{\partial k^2}{\partial w^2}$$ $$=\frac {\partial sum(k^2)}{\partial k^2}* \frac {\partial (w^2 \otimes a^1 )} {\partial w^2}$$ $$\frac{\partial z^2}{\partial w^2} = (1^{\rightarrow})^T* diag(a^1) =(a^{1})^T \quad \rightarrow (Eq \;B.3)$$ How the above is, you need to check this in https://explained.ai/matrix-calculus/#sec6.2

Basically though these are written like scalar here; actually all these are partial differention of vector by vector, or vector by scalar; and a set of vectors can be represented as the matrix here.

Note that the vector dot product $$w.a$$ when applied on matrices becomes the elementwise multiplication $$w^2 \otimes a^1$$ (also called Hadamard product)

Going back to $$Eq \;(B.2)$$

$$\frac {\partial a^2}{\partial w^2} = \frac{\partial a^2}{\partial z^2} * \frac{\partial z^2}{\partial w^2}$$

Using $$Eq \;(B.3)$$ for the term in left

$$= \frac{\partial a^2}{\partial z^2} * (a^{1})^T$$

$$= \frac{\partial \sigma(z^2)}{\partial z^2} * (a^{1})^T$$

$$\frac {\partial a^2}{\partial w^2} = \sigma^{'}(z^2) * (a^{1})^T \quad \rightarrow (Eq \;B.4)$$

Now lets got back to partial derivative of Loss function wrto to weight

$$\frac {\partial C}{\partial w^2} = \frac {1}{2}*2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B)$$ Using $$Eq \;(B.4)$$ to substitute in the last term

$$= v(0- \sigma^{'}(z^2) * (a^{1})^T)$$

$$= v*-1*\sigma^{'}(z^2) * (a^{1})^T$$

$$= (y-a^2)*-1*\sigma^{'}(z^2) * (a^{1})^T$$

$$\frac {\partial C}{\partial w^2}= (a^2-y)*\sigma^{'}(z^2) * (a^{1})^T$$