I'm having trouble wrapping my head around some details of neural nets and back prop.

For example's sake, consider the following net, where I have separated the 'neurons' into linear nodes plus activation (in this case sigmoid) nodes, more like a general computation graph. L2 is the squared loss function.

enter image description here

This is a 3 part question (parts 1 and 2 disregarding linear algebra / vectorization).

1. I want to confirm that I cannot just apply chain rule all the way from the L2 to the inputs and get the derivatives for weights at different layers.

For example:

enter image description here


enter image description here

For w5, at o1 I took the derivative with respect to w5. But for w1, at o1 I had to instead take the derivative with respect to ah1 to keep moving backwards.

2. I want to confirm that at any node that branches into more than one node at forward time (i.e. ah1), when I back prop, I have to add the derivatives of all it's branches: in this case do1/dah1 and do2/ah1.

3. If these 2 things are as I say, then how can I implement backprop in vectorized + linear algebra way without branching or using conditional logic.

The forward pass is easy:

  • x1 + x2 are a single input vector.
  • h1 + h2 are a matrix of 2 rows (number of output / units) and 2 columns (number of inputs).
  • ah1 + ah2 are a single function that can operate on vectors (i.e. 1/(1 + np.exp()))

The same for the next layer. So to do a forward pass, I just do: L2(y, a_o(W_o @ a_h(W_h @ x))) (@ = matrix mul)

But for the back pass not so much, since I'd have to check which derivative to use at o1 and I'd have to sum o1 and o2 at ah1 and ah2.

http://neuralnetworksanddeeplearning.com/chap2.html this shows that in theory it can be done by always transposing the previous weight matrix, but I haven't fully understood it yet.


So as far as I understood your problem is of mathematical nature, especially about back-propagation derivatives. Just a quick clarification, the squared loss function is not the only cost function, there might be other types of cost function. Anyways, coming to the first part of the question:

  • You can just actually apply chain rule directly from L2 to wherever you want. The weights are independent variables, they don't depend on each other. So dw5/dw1 will turn out to be 0 i.e you are differentiating a constant, in this case w5 with respect to w1. Why do we write this

enter image description here

instead of this

enter image description here

is because we can differentiate the loss function L2 w.r.t to a single independent variable only (of our choice), you cannot differentiate it w.r.t to an independent variable and still go on differentiating w.r.t to another independent variable. Its purposeless since you want to find the variation of L2 w.r.t to a particular weight only. Here are 2 links for better understanding of partial derivatives: Intuitive reasoning behind the Chain Rule in multiple variables? and Chain Rule Intuition

  • Yes, you certainly have to add the derivatives of all the branches. Basically first we look from the frame of reference of o1 and o2. From this frame o1 and o2 are independent variables and L2 is dependent on them and so we differentiate L2 w.r.t them. Then we move a step back and check what is o1 and o2 dependent on. How you go about doing this simply experience and an understanding of the thing you are doing. There is no steadfast algorithm from a simple point of view.
  • The vectorized implementation requires you follow a certain framework, it will be very lengthy here to explain the framework, but I am linking you to certain lecture videos which will make you aware of the convention being followed for vectorized implementation. Just check week 2 and week 3.

Also the thing I was talking about having different L2's in the begining is important in the sense is that in the future we may come up with weights which are dependent on other weights and so now we may have to do a dw_n/dw_m step since w_n is dependent on w_m, but lets keep that for another day. Hope this helps!

EDIT: this might give you a sense of why I am calling variables as independent variables How do I know if my back-propagation is implemented correctly?

  • $\begingroup$ thanks for the reply. I'm still confused though. So from your reply, for dL/dw5 I end the chain rule with a do1/dw5 but for dL/dw1 I skip the do1/dw5 and instead do do1/dah1 to continue propagating back? Or are you saying that both for w5 and for w1 the chain of derivatives is the same at the node in question? $\endgroup$ – SaldaVonSchwartz Mar 10 '18 at 4:17
  • $\begingroup$ @SaldaVonSchwartz ya I am saying you have to skip dw5/dw1 because they are independent variables, so w5 is not dependent on w1...its kind of like ur predicting the change of weather w.r.t to the food you eat where w5 is the change of weather and w1 is the food you eat...it doesn't make mathematical sense...also I think the part you are mistaking is the view you are giving to w5...w5 is just an attachment whereas an activation function is a tunnel...you have to go through the tunnel to reach a place...you don't have to go through w5 to reach w1..get it? the image maybe kinda misrepresenting $\endgroup$ – user9947 Mar 10 '18 at 4:43
  • $\begingroup$ yeah I see what you mean about the image, cause the weights should be more like dead end lines into the circles. But then for w5 you have to end the chain rule at do1/dw5 and for w1 you instead skip that derivative and at do1 do do1/dah1 right? I want to confirm this. The implication for my implementation is that I need conditional logic there to do one derivative or the other depending on wether I am solving for w5 or w1. Can you confirm this? thanks. $\endgroup$ – SaldaVonSchwartz Mar 10 '18 at 5:30
  • $\begingroup$ let me clarify my previous question. I am aware that w5 does not depend on w1 and that to get the partial derivative of loss for w1, w5 would be treated as constant and disappear. I'm trying to confirm that then, I cannot just apply chain rule from L2 all the way to x1 and get the partials of both w1 and w5 in one go without conditional logic. Rather, I will need to differentiate up to o1 and at that point, save do1/dw5 and then keep differentiating from do1/dah1 onwards to get to h1 and do dh1/dw1. Is this correct? This is about figuring if I can code it without testing for this condition. $\endgroup$ – SaldaVonSchwartz Mar 10 '18 at 6:09
  • $\begingroup$ @SaldaVonSchwartz yes correct....but in NN's we don't use conditional logic due to the small number of layers....I am sure you know about the nice vectorized implementations $\endgroup$ – user9947 Mar 10 '18 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.