A few doubts on back propagation

Question

I'm having trouble wrapping my head around some details of neural nets and back prop.

For example's sake, consider the following net, where I have separated the 'neurons' into linear nodes plus activation (in this case sigmoid) nodes, more like a general computation graph. L2 is the squared loss function.

This is a 3 part question (parts 1 and 2 disregarding linear algebra / vectorization).

1. I want to confirm that I cannot just apply chain rule all the way from the L2 to the inputs and get the derivatives for weights at different layers.

For example:

But

For w5, at o1 I took the derivative with respect to w5. But for w1, at o1 I had to instead take the derivative with respect to ah1 to keep moving backwards.

2. I want to confirm that at any node that branches into more than one node at forward time (i.e. ah1), when I back prop, I have to add the derivatives of all it's branches: in this case do1/dah1 and do2/ah1.

3. If these 2 things are as I say, then how can I implement backprop in vectorized + linear algebra way without branching or using conditional logic.

The forward pass is easy:

• x1 + x2 are a single input vector.
• h1 + h2 are a matrix of 2 rows (number of output / units) and 2 columns (number of inputs).
• ah1 + ah2 are a single function that can operate on vectors (i.e. 1/(1 + np.exp()))

The same for the next layer. So to do a forward pass, I just do: L2(y, a_o(W_o @ a_h(W_h @ x))) (@ = matrix mul)

But for the back pass not so much, since I'd have to check which derivative to use at o1 and I'd have to sum o1 and o2 at ah1 and ah2.

http://neuralnetworksanddeeplearning.com/chap2.html this shows that in theory it can be done by always transposing the previous weight matrix, but I haven't fully understood it yet.

Answer 1

So as far as I understood your problem is of mathematical nature, especially about back-propagation derivatives. Just a quick clarification, the squared loss function is not the only cost function, there might be other types of cost function. Anyways, coming to the first part of the question:

• You can just actually apply chain rule directly from L2 to wherever you want. The weights are independent variables, they don't depend on each other. So dw5/dw1 will turn out to be 0 i.e you are differentiating a constant, in this case w5 with respect to w1. Why do we write this

instead of this

is because we can differentiate the loss function L2 w.r.t to a single independent variable only (of our choice), you cannot differentiate it w.r.t to an independent variable and still go on differentiating w.r.t to another independent variable. Its purposeless since you want to find the variation of L2 w.r.t to a particular weight only. Here are 2 links for better understanding of partial derivatives: Intuitive reasoning behind the Chain Rule in multiple variables? and Chain Rule Intuition

• Yes, you certainly have to add the derivatives of all the branches. Basically first we look from the frame of reference of o1 and o2. From this frame o1 and o2 are independent variables and L2 is dependent on them and so we differentiate L2 w.r.t them. Then we move a step back and check what is o1 and o2 dependent on. How you go about doing this simply experience and an understanding of the thing you are doing. There is no steadfast algorithm from a simple point of view.
• The vectorized implementation requires you follow a certain framework, it will be very lengthy here to explain the framework, but I am linking you to certain lecture videos which will make you aware of the convention being followed for vectorized implementation. Just check week 2 and week 3.

Also the thing I was talking about having different L2's in the begining is important in the sense is that in the future we may come up with weights which are dependent on other weights and so now we may have to do a dw_n/dw_m step since w_n is dependent on w_m, but lets keep that for another day. Hope this helps!

EDIT: this might give you a sense of why I am calling variables as independent variables How do I know if my back-propagation is implemented correctly?