I'm having trouble wrapping my head around some details of neural nets and back prop.
For example's sake, consider the following net, where I have separated the 'neurons' into linear nodes plus activation (in this case sigmoid) nodes, more like a general computation graph. L2 is the squared loss function.
This is a 3 part question (parts 1 and 2 disregarding linear algebra / vectorization).
1. I want to confirm that I cannot just apply chain rule all the way from the L2 to the inputs and get the derivatives for weights at different layers.
For w5, at o1 I took the derivative with respect to w5. But for w1, at o1 I had to instead take the derivative with respect to ah1 to keep moving backwards.
2. I want to confirm that at any node that branches into more than one node at forward time (i.e. ah1), when I back prop, I have to add the derivatives of all it's branches: in this case do1/dah1 and do2/ah1.
3. If these 2 things are as I say, then how can I implement backprop in vectorized + linear algebra way without branching or using conditional logic.
The forward pass is easy:
- x1 + x2 are a single input vector.
- h1 + h2 are a matrix of 2 rows (number of output / units) and 2 columns (number of inputs).
- ah1 + ah2 are a single function that can operate on vectors (i.e. 1/(1 + np.exp()))
The same for the next layer. So to do a forward pass, I just do: L2(y, a_o(W_o @ a_h(W_h @ x))) (@ = matrix mul)
But for the back pass not so much, since I'd have to check which derivative to use at o1 and I'd have to sum o1 and o2 at ah1 and ah2.
http://neuralnetworksanddeeplearning.com/chap2.html this shows that in theory it can be done by always transposing the previous weight matrix, but I haven't fully understood it yet.