I am trying to understand backpropagation. I used a simple neural network with one input x, one hidden layer h and one output layer y, with weight w1 connecting x to h, and w2 connecting h to y.

x--[w1]--> h --[w2]-->y

In my understanding these are the steps happening while we train a neural network:

enter image description here

I understood most parts of backpropogation, but how do we get the gradients for the middle layer weights dL/dw1?



Feed \ forwarding \\
h=\sigma (xw_{1}+b) \\ 
{y}'=\sigma (hw_{2}+b) \\ \\
Loss \ function \\ 
L=\frac{1}{2}\sum(y-{y}')^{2} \\ \\ 
Gradient \ calculation \\ \\
\frac{\partial L}{\partial w_{2}}=\frac{\partial {y}'}{\partial w_{2}}\frac{\partial L }{\partial {y}'} \\ \\ 
\frac{\partial L}{\partial w_{1}}= \frac{\partial h}{\partial w_{1}} \frac{\partial {y}'}{\partial h} \frac{\partial L}{\partial {y}'}   \\ \\ % DuttaA's solution
Weight \ update \\ 
w_{i}^{t+1} \leftarrow w_{i}^{t}-\alpha \frac{\partial L}{\partial w_{i}}

How should we calculate gradient of a network similar to this? enter image description here

is this the correct equation?

enter image description here

Latex format

\frac{\partial L}{\partial w_{1}}=\frac{\partial h_{1}}{\partial w_{1}}\frac{\partial w_7}{\partial h_{1}}\frac{\partial o_2}{\partial w_{7}}\frac{\partial L}{\partial o_{2}}  + \frac{\partial h_{1}}{\partial w_{1}}\frac{\partial w_5}{\partial h_{1}}\frac{\partial o_1}{\partial w_{5}}\frac{\partial L}{\partial o_{1}}
  • 3
    $\begingroup$ (dl/dy)*(dy/dh)*(dh/dw1) also called the chain rule...math.stackexchange.com/questions/62614/… and math.stackexchange.com/questions/423/… $\endgroup$ – DuttaA Mar 8 '18 at 16:37
  • $\begingroup$ First link is 404. Other one is great. $\endgroup$ – SherylHohman Mar 9 '18 at 1:25
  • 1
    $\begingroup$ @SherylHohman math.stackexchange.com/questions/62614/chain-rule-intuition i was having prblm posting multiple links $\endgroup$ – DuttaA Mar 9 '18 at 2:53
  • $\begingroup$ @DuttaA I have edited the question and added some additonal doubts to my question. Can you look into it and check if its correct or not. Please write as an answer so that I can upvote and accept it including my first unedited question. $\endgroup$ – Eka Mar 10 '18 at 14:41
  • $\begingroup$ @Eka sorry for the long answer...I am working towards the general goal of making this site more accessible to beginners..u'll find your answer below the pic $\endgroup$ – DuttaA Mar 10 '18 at 15:52

So, the main doubt here is about the intuition behind the derivative part of back-propagation learning. First, I would like to point out 2 links about the intuition about how partial derivatives work Chain Rule Intuition and Intuitive reasoning behind the Chain Rule in multiple variables?.

Now that we know how the chain rule works, lets see how we can use it in Machine Learning. So basically in machine learning the final output is a function of input variables and the connection weights f(x_1, x_2...x_n, w_1, w_2...w_n) where f encloses all the activation functions and dot products lying between input and output. The x_1, x_2...x_n, w_1, w_2...w_n are called independent variables because they don't affect each other pairwise as well as in groups meaning you cannot find a function g(x_i..., w_i...) = h(x_j...,w_j..) So basically its a black box from input to output.

enter image description here

So now our purpose is to minimize the Loss/Cost function, by changing the parameters that can be 'controlled by us' i.e the weights only, we cannot change the input variables. So this is done by taking the derivative of the cost function w.r.t to the variable that 'can be changed'. Here is an explanation of why taking derivative and subsequently subtracting it reduces the value of cost function given by 'maximal' amount. Also here.

Now, to calculate dL/dw_n you have to keep few things in mind:

  • Only differentiate L w.r.t to those functions which affect L.
  • And to reach to your end goal of differentiation w.r.t to an independent variable you must differentiate L w.r.t to those functions only which are dependent on that particular independent variable.

A crude algorithm assuming 'L' also as a normal function (along the lines of activation function, so that I can express the idea recursively) differentiate f_n w.r.t to functions in the previous layer say f_n-1, f_n-2, w_n. Check which of these functions depend on w_1. Only f_n-1 and f_n-2 do. Differentiate them again w.r.t to previous layer functions. Check again and go on till you reach w_1.

This approach is the fool-proof version, but it has 2 flaws:

  • First, w_n is not a function. People are making this mistake of assuming w_n to be a function due to misinterpretation of a simple NN diagram. To reachw_1 you don't need to go through w_n. But you definitely need to go through the activation functions and dot products. Think of this as painting a wall where color mixing occurs (not over-writing). So you paint the wall with some color (weights) then 2nd color and so on. Is the final product affected by color 1. Yes. Is the 'rate of change' caused by color 1 also affected by color 2. Yes. But does it mean we can find the 'change'of color n w.r.t to color 1? No its meaningless (bad example, couldn't think of a better one)
  • The second flaw is that this approach is not followed because with experience it is apparent which function affects whom and which independent variable affects which function (saves computation).

To answer your question the equation is incorrect and the correct equation will be:

enter image description here

I have simply followed the algorithm I have given above.

As for why your equation is wrong, your equation contains the term dw7/dh1. Does w7 vary with h1? This means that w7 is directly related to the input as h1 is related with the input, but this is not the case for a single iteration(the whole algorithm run makes wn dependent on the inputs since you are trying to minimize the loss function w.r.t given inputs and weights, for a different set of inputs you will have different final weights).

So in a nutshell, the aim of back-propagation is to identify the change in Loss function w.r.t to a given weights. To calculate that you have to make sure in the chain rule of derivative you don't have any meaningless terms like derivative of an independent variable w.r.t to any function. I recommend checking Khan Academy for a better understanding and clarity in concepts as I think the intuitions are hard to provide in a written answer.

  • $\begingroup$ Thanks I got it w7 and w5 are constants whereas h1 and o1 are functions. w7 and w5 only changes during updation whereas h1 and o1 changes in forward passing aswell $\endgroup$ – Eka Mar 10 '18 at 16:13
  • 1
    $\begingroup$ @Eka good..I missed that part but you got it $\endgroup$ – DuttaA Mar 10 '18 at 16:16
  • $\begingroup$ One other question; so when we are finding gradients with respect to a weight. We keep that weight as variable and the rest of weights as constant. in my above case w1 is variable and the other two constant.. right? $\endgroup$ – Eka Mar 10 '18 at 16:28
  • 1
    $\begingroup$ @Eka yes...that is pretty much the concept of how partial derivatives are calculated $\endgroup$ – DuttaA Mar 10 '18 at 16:38

I'd recommend studying a bit of calculus, and reading about the chain rule.

Short Explanation:

we have f(y) and y(x)
df/dx = df/dy * dy/dx
this is the chain rule, it can be applied many times.
if we have f(y), y(z), z(x)
df/xd = df/dy * dy/dx
dy/dx = dy/dz * dz/dx
so df/xd = df/dy * dy/dz * dz/dx chain!! :)

for your example:
dl/dw1 = dl/dy' * dy'/dh * dh/dw1


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.