2
$\begingroup$

While looking at the mathematics of the back-propagation algorithm for a multi-layer perceptron, I noticed that in order to find the partial derivative of the cost function with respect to a weight (say $w$) from any of the hidden layers, we're just writing the error function from the final outputs in terms of the inputs and hidden layer weights and then canceling all the terms without $w$ in it as differentiating those terms with respect to $w$ would give zero.

Where is the back-propagation of error while doing this? This way, I can find the partial derivatives of the first hidden layer first and then go towards the other ones if I wanted to. Is there some other method of going about it so that the Back Propagation concept comes into play? Also, I'm looking for a general method/algorithm, not just for 1-2 hidden layers.

I'm fairly new to this and I'm just following what's being taught in class. Nothing I found on the internet seems to have proper notation so I can't understand what they're saying.

$\endgroup$
1
$\begingroup$

Have a look at the following article Principles of training multi-layer neural network using backpropagation. It was very useful to me.

You can also see here an example of backpropagation in Matlab. It effectively solves the XOR problem. You can also play around with the cost function or the learning rate. You may get surprising results! Does this answer your question?

| improve this answer | |
$\endgroup$
  • $\begingroup$ I don't see how the method in the link you shared would work if there are activation functions on each node, it seems to give a different result to the one I referred to in the question. It makes sense if there are no activation functions though. Can you explain how the error gets propagated backward the same way the input is propagated forward? Thanks $\endgroup$ – Skawang Apr 10 at 11:45
  • $\begingroup$ You need to find the derivative of the function e.g sigmoid'(e) = s(e)*(1-s(e)) $\endgroup$ – david david Apr 10 at 16:43
4
$\begingroup$

Why is it called back-propagation?

I don't think there is anything special here!

It's called back-propagation (BP) because, after the forward pass, you compute the partial derivative of the loss function with respect to the parameters of the network, which, in the usual diagrams of a neural network, are placed before the output of the network (i.e. to the left of the output if the output of the network is on the right, or to the right if the output of the network is on the left).

It's also called BP because it is just the application of the chain rule. Why is this interesting?

Let me answer this question with an example. Consider the function $y=e^{\sin(x^{2})}$. This is a composite function, i.e. a function composed of multiple simpler functions, which, in this case, are $e^x$, $\sin(x)$, $x^2$ and $x$. To compute the derivative of $y$ with respect to $x$, let's define the following variables

\begin{align} y &= f(u) = e^u,\\ u &= g(v) = \sin v = \sin(x^2),\\ v &= h(x) = x^2 \end{align}

The derivative of $y$ with respect to the variable $x$ is (according to the chain rule)

$$ \underset{\color{red}{\LARGE \rightarrow}}{ \frac{dy}{dx} = \frac{dy}{du} \color{green}{\cdot} \frac{du}{dv} \color{green}{\cdot} \frac{dv}{dx}} $$

If you read this equation from the left to the right, you can see that we are going backward (i.e. from the function $y$ to the function $v$). This is the same thing with BP!

Why is it called "chain rule"? Because you are chaining different partial derivatives. More specifically, you are multiplying them.

BP is also known as the reverse mode of automatic differentiation. Why? The automatic differentiation should be self-explanatory, given that the BP algorithm is just the computation of partial derivatives, and you do this automatically, i.e. with a program, rather than by hand. The expression "reverse mode" refers to the fact that we compute the derivatives from the outer function (which, in the example above, is $e^x$) to the inner function (which, in the example above, is $x$). The Wikipedia article related to automatic differentiation provides more details.

What exactly are you back-propagating?

The partial derivative of the loss function $\mathcal{L}$ with respect to a parameter $w_i$, i.e. $\frac{\partial \mathcal{L}}{\partial w_i}$, intuitively, represents the "contribution" of the parameter $w_i$ to the loss. After having computed these partial derivatives (i.e. the gradient), you use gradient descent to update each parameter $w_i$ as follows

$$ w_i \leftarrow w_i - \gamma \frac{\partial \mathcal{L}}{\partial w_i} $$

where $\frac{\partial \mathcal{L}}{\partial w_i}$ represents WHAT we propagatED, which is the error (or loss) that the neural network makes.

This gradient descent step will hopefully make your network produce a smaller error next time.

The modern version of back-propagation was published (in 1970) by a Finnish master's student called Seppo Linnainmaa, although some people claim this is not the case because they want to give credits to others (because Seppo Linnainmaa didn't reference neural networks, but back-propagation isn't only applicable to neural networks) or they don't even know the history. This Jürgen Schmidhuber's article goes into the details of the history of BP.

| improve this answer | |
$\endgroup$
  • $\begingroup$ So you're saying the only reason it's called back propagation is because of the chain rule? I was under the impression that there is some kind of propagation of error from the output to the input, like you calculate the error on the final hidden layer, then use that to find the error in the layer before that and so on, kind of like how you propagate the input to hidden layer to output. Thanks for the answer. $\endgroup$ – Skawang Apr 10 at 11:47
  • $\begingroup$ @Skawang Yes, you can view the chain rule also in that way, but that's still the chain rule, I just didn't explicitly show the "error" and how it is part of the equations, but you can find this online. $\endgroup$ – nbro Apr 10 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.