How is the gradient calculated for the middle layer's weights?

Question

I am trying to understand backpropagation. I used a simple neural network with one input $x$, one hidden layer $h$ and one output layer $y$, with weight $w_1$ connecting $x$ to $h$, and $w_2$ connecting $h$ to $y$

$$ x \rightarrow (w_1) \rightarrow h \rightarrow (w_2) \rightarrow y $$

In my understanding, these are the steps happening while we train a neural network:

The feedforward step.

\begin{align} h=\sigma\left(x w_{1}+b\right)\\ y^{\prime}=\sigma\left(h w_{2}+b\right) \end{align}

The loss function.

$$ L=\frac{1}{2} \sum\left(y-y^{\prime}\right)^{2} $$

The gradient calculation

$$\frac{\partial L}{\partial w_{2}}=\frac{\partial y^{\prime}}{\partial w_{2}} \frac{\partial L}{\partial y^{\prime}}$$

$$\frac{\partial L}{\partial w_{1}}=?$$

The weight update

$$ w_{i}^{t+1} \leftarrow w_{i}^{t}-\alpha \frac{\partial L}{\partial w_{i}} $$

I understood most parts of backpropagation, but how do we get the gradients for the middle layer weights $dL/dw_1$?

How should we calculate the gradient of a network similar to this?

Is this the correct equation?

$$\frac{\partial L}{\partial w_{1}}=\frac{\partial h_{1}}{\partial w_{1}} \frac{\partial w_{7}}{\partial h_{1}} \frac{\partial o_{2}}{\partial w_{7}} \frac{\partial L}{\partial o_{2}}+\frac{\partial h_{1}}{\partial w_{1}} \frac{\partial w_{5}}{\partial h_{1}} \frac{\partial o_{1}}{\partial w_{5}} \frac{\partial L}{\partial o_{1}}$$

Answer 1

The main doubt here is about the intuition behind the derivative part of back-propagation learning. First, I would like to point out 2 links about the intuition about how partial derivatives work Chain Rule Intuition and Intuitive reasoning behind the Chain Rule in multiple variables?.

Now that we know how the chain rule works, let's see how we can use it in Machine Learning. So basically in machine learning, the final output is a function of input variables and the connection weights $f\left(x_1, x_2, \ldots, x_n, w_1, w_2, \ldots, w_n\right)$, where $f$ encloses all the activation functions and dot products lying between input and output. The $x_1, x_2, \ldots, x_n, w_1, w_2, \ldots, w_n$ are called independent variables because they don't affect each other pairwise as well as in groups meaning you cannot find a function $g(x_i, \dots, w_i, \dots) = h(x_j, \dots, w_j, \dots)$. So, basically its a black box from input to output.

So now our purpose is to minimize the Loss/Cost function, by changing the parameters that can be 'controlled by us' i.e the weights only, we cannot change the input variables. So this is done by taking the derivative of the cost function w.r.t to the variable that 'can be changed'. Here is an explanation of why taking derivative and subsequently subtracting it reduces the value of cost function given by the 'maximal' amount. Also here.

Now, to calculate $dL/dw_n$ you have to keep few things in mind:

• Only differentiate $L$ w.r.t to those functions which affect $L$.
• And to reach your end goal of differentiation w.r.t to an independent variable you must differentiate $L$ w.r.t to those functions only which are dependent on that particular independent variable.

A crude algorithm assuming $L$ also as a normal function (along the lines of activation function, so that I can express the idea recursively) differentiate $f_n$ w.r.t to functions in the previous layer say $f_{n-1}$, $f_{n-2}$, $w_n$. Check which of these functions depends on $w_1$. Only $f_{n-1}$ and $f_{n-2}$ do. Differentiate them again w.r.t to previous layer functions. Check again and go on till you reach $w_1$.

This approach is the fool-proof version, but it has 2 flaws:

• First, $w_n$ is not a function. People are making this mistake of assuming $w_n$ to be a function due to misinterpretation of a simple NN diagram. To reach $w_1$, you don't need to go through $w_n$. But you definitely need to go through the activation functions and dot products. Think of this as painting a wall where color mixing occurs (not over-writing). So you paint the wall with some color (weights) then 2nd color and so on. Is the final product affected by color 1. Yes. Is the 'rate of change' caused by color 1 also affected by color 2. Yes. But does it mean we can find the 'change'of color n w.r.t to color 1? No its meaningless (bad example, couldn't think of a better one)
• The second flaw is that this approach is not followed because with experience it is apparent which function affects whom and which independent variable affects which function (saves computation).

To answer your question the equation is incorrect and the correct equation will be:

$$ \frac{\partial L}{\partial w_{1}}=\frac{\partial h_{1}}{\partial w_{1}} \frac{\partial o_{2}}{\partial h_{1}} \frac{\partial L}{\partial o_{2}}+\frac{\partial h_{1}}{\partial w_{1}} \frac{\partial o_{1}}{\partial h_{1}} \frac{\partial L}{\partial o_{1}} $$

I have simply followed the algorithm I have given above.

As for why your equation is wrong, your equation contains the term $dw7/dh1$. Does $w_7$ vary with $h_1$? This means that $w_7$ is directly related to the input as $h_1$ is related with the input, but this is not the case for a single iteration(the whole algorithm run makes $w_n$ dependent on the inputs since you are trying to minimize the loss function w.r.t given inputs and weights, for a different set of inputs you will have different final weights).

So, in a nutshell, the aim of back-propagation is to identify the change in the loss function w.r.t to a given weight. To calculate that, you have to make sure in the chain rule of derivative you don't have any meaningless terms like the derivative of an independent variable w.r.t to any function. I recommend checking Khan Academy for a better understanding and clarity in concepts as I think the intuitions are hard to provide in a written answer.