So, the main doubt here is about the intuition behind the derivative part of back-propagation learning. First, I would like to point out 2 links about the intuition about how partial derivatives work Chain Rule Intuition and Intuitive reasoning behind the Chain Rule in multiple variables?.
Now that we know how the chain rule works, lets see how we can use it in Machine Learning. So basically in machine learning the final output is a function of input variables and the connection weights
f(x_1, x_2...x_n, w_1, w_2...w_n) where
f encloses all the activation functions and dot products lying between input and output. The
x_1, x_2...x_n, w_1, w_2...w_n are called independent variables because they don't affect each other pairwise as well as in groups meaning you cannot find a function
g(x_i..., w_i...) = h(x_j...,w_j..) So basically its a black box from input to output.
So now our purpose is to minimize the Loss/Cost function, by changing the parameters that can be 'controlled by us' i.e the weights only, we cannot change the input variables. So this is done by taking the derivative of the cost function w.r.t to the variable that 'can be changed'. Here is an explanation of why taking derivative and subsequently subtracting it reduces the value of cost function given by 'maximal' amount. Also here.
Now, to calculate
dL/dw_n you have to keep few things in mind:
- Only differentiate
L w.r.t to those functions which affect
- And to reach to your end goal of differentiation w.r.t to an independent variable you must differentiate
L w.r.t to those functions only which are dependent on that particular independent variable.
A crude algorithm assuming 'L' also as a normal function (along the lines of activation function, so that I can express the idea recursively) differentiate
f_n w.r.t to functions in the previous layer say
f_n-1, f_n-2, w_n. Check which of these functions depend on
f_n-2 do. Differentiate them again w.r.t to previous layer functions. Check again and go on till you reach
This approach is the fool-proof version, but it has 2 flaws:
w_n is not a function. People are making this mistake of assuming
w_n to be a function due to misinterpretation of a simple NN diagram. To reach
w_1 you don't need to go through
w_n. But you definitely need to go through the activation functions and dot products. Think of this as painting a wall where color mixing occurs (not over-writing). So you paint the wall with some color (weights) then 2nd color and so on. Is the final product affected by color 1. Yes. Is the 'rate of change' caused by color 1 also affected by color 2. Yes. But does it mean we can find the 'change'of color n w.r.t to color 1? No its meaningless (bad example, couldn't think of a better one)
- The second flaw is that this approach is not followed because with experience it is apparent which function affects whom and which independent variable affects which function (saves computation).
To answer your question the equation is incorrect and the correct equation will be:
I have simply followed the algorithm I have given above.
As for why your equation is wrong, your equation contains the term
w7 vary with
h1? This means that
w7 is directly related to the input as
h1 is related with the input, but this is not the case for a single iteration(the whole algorithm run makes
wn dependent on the inputs since you are trying to minimize the loss function w.r.t given inputs and weights, for a different set of inputs you will have different final weights).
So in a nutshell, the aim of back-propagation is to identify the change in Loss function w.r.t to a given weights. To calculate that you have to make sure in the chain rule of derivative you don't have any meaningless terms like derivative of an independent variable w.r.t to any function. I recommend checking Khan Academy for a better understanding and clarity in concepts as I think the intuitions are hard to provide in a written answer.