# Why is it called back-propagation?

While looking at the mathematics of the back-propagation algorithm for a multi-layer perceptron, I noticed that in order to find the partial derivative of the cost function with respect to a weight (say $$w$$) from any of the hidden layers, we're just writing the error function from the final outputs in terms of the inputs and hidden layer weights and then canceling all the terms without $$w$$ in it as differentiating those terms with respect to $$w$$ would give zero.

Where is the back-propagation of error while doing this? This way, I can find the partial derivatives of the first hidden layer first and then go towards the other ones if I wanted to. Is there some other method of going about it so that the Back Propagation concept comes into play? Also, I'm looking for a general method/algorithm, not just for 1-2 hidden layers.

I'm fairly new to this and I'm just following what's being taught in class. Nothing I found on the internet seems to have proper notation so I can't understand what they're saying.

Have a look at the following article Principles of training multi-layer neural network using backpropagation. It was very useful to me.

You can also see here an example of backpropagation in Matlab. It effectively solves the XOR problem. You can also play around with the cost function or the learning rate. You may get surprising results! Does this answer your question?

• I don't see how the method in the link you shared would work if there are activation functions on each node, it seems to give a different result to the one I referred to in the question. It makes sense if there are no activation functions though. Can you explain how the error gets propagated backward the same way the input is propagated forward? Thanks Apr 10 '20 at 11:45
• You need to find the derivative of the function e.g $\text{sigmoid}'(e) = s(e)*(1-s(e))$ Apr 10 '20 at 16:43

Why is it called back-propagation?

I don't think there is anything special here!

It's called back-propagation (BP) because, after the forward pass, you compute the partial derivative of the loss function with respect to the parameters of the network, which, in the usual diagrams of a neural network, are placed before the output of the network (i.e. to the left of the output if the output of the network is on the right, or to the right if the output of the network is on the left).

It's also called BP because it is just the application of the chain rule. Why is this interesting?

Let me answer this question with an example. Consider the function $$y=e^{\sin(x^{2})}$$. This is a composite function, i.e. a function composed of multiple simpler functions, which, in this case, are $$e^x$$, $$\sin(x)$$, $$x^2$$ and $$x$$. To compute the derivative of $$y$$ with respect to $$x$$, let's define the following variables

\begin{align} y &= f(u) = e^u,\\ u &= g(v) = \sin v = \sin(x^2),\\ v &= h(x) = x^2 \end{align}

The derivative of $$y$$ with respect to the variable $$x$$ is (according to the chain rule)

$$\underset{\color{red}{\LARGE \rightarrow}}{ \frac{dy}{dx} = \frac{dy}{du} \color{green}{\cdot} \frac{du}{dv} \color{green}{\cdot} \frac{dv}{dx}}$$

If you read this equation from the left to the right, you can see that we are going backward (i.e. from the function $$y$$ to the function $$v$$). This is the same thing with BP!

Why is it called "chain rule"? Because you are chaining different partial derivatives. More specifically, you are multiplying them.

BP is also known as the reverse mode of automatic differentiation. Why? The automatic differentiation should be self-explanatory, given that the BP algorithm is just the computation of partial derivatives, and you do this automatically, i.e. with a program, rather than by hand. The expression "reverse mode" refers to the fact that we compute the derivatives from the outer function (which, in the example above, is $$e^x$$) to the inner function (which, in the example above, is $$x$$). The Wikipedia article related to automatic differentiation provides more details.

What exactly are you back-propagating?

The partial derivative of the loss function $$\mathcal{L}$$ with respect to a parameter $$w_i$$, i.e. $$\frac{\partial \mathcal{L}}{\partial w_i}$$, intuitively, represents the "contribution" of the parameter $$w_i$$ to the loss. After having computed these partial derivatives (i.e. the gradient), you use gradient descent to update each parameter $$w_i$$ as follows

$$w_i \leftarrow w_i - \gamma \frac{\partial \mathcal{L}}{\partial w_i}$$

where $$\frac{\partial \mathcal{L}}{\partial w_i}$$ represents WHAT we propagatED, which is the error (or loss) that the neural network makes.

This gradient descent step will hopefully make your network produce a smaller error next time.

The modern version of back-propagation was published (in 1970) by a Finnish master's student called Seppo Linnainmaa, although some people claim this is not the case because they want to give credits to others (because Seppo Linnainmaa didn't reference neural networks, but back-propagation isn't only applicable to neural networks) or they don't even know the history. This Jürgen Schmidhuber's article goes into the details of the history of BP.

• So you're saying the only reason it's called back propagation is because of the chain rule? I was under the impression that there is some kind of propagation of error from the output to the input, like you calculate the error on the final hidden layer, then use that to find the error in the layer before that and so on, kind of like how you propagate the input to hidden layer to output. Thanks for the answer. Apr 10 '20 at 11:47
• @Skawang Yes, you can view the chain rule also in that way, but that's still the chain rule, I just didn't explicitly show the "error" and how it is part of the equations, but you can find this online.
– nbro
Apr 10 '20 at 12:28
• @nbro, when using chain rule for BP in deep learning, do u agree that we can start computing the gradient (i.e. the product of some firsts layer jacobians) even if the forward pass is not terminated ? I mean one can imagine an algorithm computing both forward and backward at almost the same time ? May 20 at 13:25
• @RémyHosseinkhanBoucher You can compute the general formulas to compute the partial derivatives without doing the forward pass (as I did in my answer, for example), but you need the specific values obtained during the forward pass to compute the actual values of the partial derivatives, otherwise, what would be the purpose of the forward pass?
– nbro
May 20 at 13:51
• So, no, you need the forward pass to compute the actual gradient. You can do this exercise with a simple neural network. Maybe a 1-hidden layer net. Compute the formulas of the backprop for it. Then you will realize that to compute the actual values you need the values obtained during the forward pass.
– nbro
May 20 at 13:56