In gradient descent's update rule, why do we use $\sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}$ instead of $\frac{\delta C_0}{\delta w^{l}}$?

Question

I am trying to code a two layered neural network simple NN as I have described here https://itisexplained.com/html/NN/ml/5_codingneuralnetwork/

I am getting stuck on the last step of updating the weights after calculating the gradients for the outer and inner layers via back-propagation

#---------------------------------------------------------------

# Two layered NW. Using from (1) and the equations we derived as explanations
# (1) http://iamtrask.github.io/2015/07/12/basic-python-network/
#---------------------------------------------------------------

import numpy as np
# seed random numbers to make calculation deterministic 
np.random.seed(1)

# pretty print numpy array
np.set_printoptions(formatter={'float': '{: 0.3f}'.format})

# let us code our sigmoid funciton
def sigmoid(x):
    return 1/(1+np.exp(-x))

# let us add a method that takes the derivative of x as well
def derv_sigmoid(x):
   return x*(1-x)

# set learning rate as 1 for this toy example
learningRate =  1

# input x, also used as the training set here
x = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1]  ])

# desired output for each of the training set above
y = np.array([[0,1,1,0]]).T

# Explanaiton - as long as input has two ones, but not three, ouput is One
"""
Input [0,0,1]  Output = 0
Input [0,1,1]  Output = 1
Input [1,0,1]  Output = 1
Input [1,1,1]  Output = 0
"""

input_rows = 4
# Randomly initalised weights
weight1 =  np.random.random((3,input_rows))
weight2 =  np.random.random((input_rows,1))

print("Shape weight1",np.shape(weight1)) #debug
print("Shape weight2",np.shape(weight2)) #debug

# Activation to layer 0 is taken as input x
a0 = x

iterations = 1000
for iter in range(0,iterations):

  # Forward pass - Straight Forward
  z1= x @ weight1
  a1 = sigmoid(z1) 
  z2= a1 @ weight2
  a2 = sigmoid(z2) 

  # Backward Pass - Backpropagation 
  delta2  = (y-a2)
  #---------------------------------------------------------------
  # Calcluating change of Cost/Loss wrto weight of 2nd/last layer
  # Eq (A) ---> dC_dw2 = delta2*derv_sigmoid(z2)
  #---------------------------------------------------------------

  dC_dw2  = delta2 * derv_sigmoid(a2)

  if iter == 0:
    print("Shape dC_dw2",np.shape(dC_dw2)) #debug
  
  #---------------------------------------------------------------
  # Calcluating change of Cost/Loss wrto weight of 2nd/last layer
  # Eq (B)---> dC_dw1 = derv_sigmoid(a1)*delta2*derv_sigmoid(a2)*weight2
  # note  delta2*derv_sigmoid(a2) == dC_dw2 
  # dC_dw1 = derv_sigmoid(a1)*dC_dw2*weight2
  #---------------------------------------------------------------
  
  dC_dw1 =  (np.multiply(dC_dw2,weight2.T)) * derv_sigmoid(a1)
  if iter == 0:
    print("Shape dC_dw1",np.shape(dC_dw1)) #debug
  

  #---------------------------------------------------------------
  #Gradinent descent
  #---------------------------------------------------------------
 
  #weight2 = weight2 - learningRate*dC_dw2 --> these are what the textbook tells
  #weight1 = weight1 - learningRate*dC_dw1 

  weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2) # this is what works
  weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1) 
  

print("New ouput\n",a2)

Why is

  weight2 = weight2 + learningRate*np.dot(a1.T,dC_dw2)
  weight1 = weight1 + learningRate*np.dot(a0.T,dC_dw1) 

done instead of

  #weight2 = weight2 - learningRate*dC_dw2
  #weight1 = weight1 - learningRate*dC_dw1 

I am not getting the source of the equation of updating the weights by multiplying with the activation of the previous layer

As per gradient descent, the weight update should be

$$ W^{l}_{new} = W^{l}_{old} - \gamma * \frac{\delta C_0}{\delta w^{l}} $$

However, what works in practice is

$$ W^{l}_{new} = W^{l}_{old} - \gamma * \sigma(z^{l-1})\frac{\delta C_0}{ \delta w^{l}}, $$

where $\gamma$ is the learning rate.

Answer 1

Okay - the answer is here https://explained.ai/matrix-calculus/#sec6.2 and it is pretty involved. Basically, there is a difference when you derive the equation for one neuron and when you have to do practically for a set of neurons. The answer is matrix calculus. Here goes from what I could make out. Feel free to correct if I am wrong

Gradient Vector/Matrix/2D tensor of Loss function wrto Weight

$$ C = \frac{1}{2} \sum_j (y_j-a^L_j)^2 $$

Assuming a neural net with 2 layers, we have the final Loss as

$$ C = \frac{1}{2} \sum_j (y_j-a^2_j)^2 $$

Where

$$ a^2 = \sigma(w^2.a^1) $$

We can then write

$$ C = \frac{1}{2} \sum_j v^2 \quad \rightarrow (Eq \;A) $$

Where

$$ v= y-a^2 $$

Partial Derivative of Loss function wrto Weight

For the last layer, lets use Chain Rule to split like below

$$ \frac {\partial C}{\partial w^2} = \frac{\partial v^2}{\partial v} * \frac{\partial v}{\partial w^2} \quad \rightarrow (Eq \;B) $$

$$ \frac{\partial v^2}{\partial v} =2v \quad \rightarrow (Eq \;B.1) $$

$$ \frac{\partial v}{\partial w^2}= \frac{\partial (y-a^2)}{\partial w^2} = 0-\frac{\partial a^2}{\partial w^2} \quad \rightarrow (Eq \;B.2) $$

$$ \frac {\partial C}{\partial w^2} = \frac{1}{2} *2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B) $$  

Now we need to find $\frac{\partial a^2}{\partial w^2}$

Let

$$ a^2= \sigma(sum(w^2 \otimes a^1 )) = \sigma(z^2) $$ $$ z^2 = sum(w^2 \otimes a^1 ) $$

$$ z^2 = sum(k^2) \; \text {where} \; k^2=w^2 \otimes a^1 $$

We now need to derive an intermediate term which we will use later

$$ \frac{\partial z^2}{\partial w^2} =\frac{\partial z^2}{\partial k^2}*\frac{\partial k^2}{\partial w^2} $$ $$ =\frac {\partial sum(k^2)}{\partial k^2}* \frac {\partial (w^2 \otimes a^1 )} {\partial w^2} $$ $$ \frac{\partial z^2}{\partial w^2} = (1^{\rightarrow})^T* diag(a^1) =(a^{1})^T \quad \rightarrow (Eq \;B.3) $$ How the above is, you need to check this in https://explained.ai/matrix-calculus/#sec6.2

Basically though these are written like scalar here; actually all these are partial differention of vector by vector, or vector by scalar; and a set of vectors can be represented as the matrix here.

Note that the vector dot product $w.a$ when applied on matrices becomes the elementwise multiplication $w^2 \otimes a^1$ (also called Hadamard product)

Going back to $Eq \;(B.2)$

$$ \frac {\partial a^2}{\partial w^2} = \frac{\partial a^2}{\partial z^2} * \frac{\partial z^2}{\partial w^2} $$

Using $Eq \;(B.3)$ for the term in left

$$ = \frac{\partial a^2}{\partial z^2} * (a^{1})^T $$

$$ = \frac{\partial \sigma(z^2)}{\partial z^2} * (a^{1})^T $$

$$ \frac {\partial a^2}{\partial w^2} = \sigma^{'}(z^2) * (a^{1})^T \quad \rightarrow (Eq \;B.4) $$

Now lets got back to partial derivative of Loss function wrto to weight

$$ \frac {\partial C}{\partial w^2} = \frac {1}{2}*2v(0-\frac{\partial a^2}{\partial w^2}) \quad \rightarrow (Eq \;B) $$ Using $Eq \;(B.4)$ to substitute in the last term

$$ = v(0- \sigma^{'}(z^2) * (a^{1})^T) $$

$$ = v*-1*\sigma^{'}(z^2) * (a^{1})^T $$

$$ = (y-a^2)*-1*\sigma^{'}(z^2) * (a^{1})^T $$

$$ \frac {\partial C}{\partial w^2}= (a^2-y)*\sigma^{'}(z^2) * (a^{1})^T $$