I'm following the guide as outlined at this link: http://neuralnetworksanddeeplearning.com/chap2.html

For the purposes of this question, I've written a basic network 2 hidden layers, one with 2 neurons and one with one neuron. For a very basic task, the network will learn how to compute an OR logic gate so the training data will be:

X = [[0, 0], [0, 1], [1, 0], [1, 1]]
Y = [0, 1, 1, 1]

And the diagram: enter image description here

For this example, the weights and biases are:

w = [[0.3, 0.4], [0.1]]
b = [[1, 1], [1]]

The feedforward part was pretty easy to implement so I don't think I need to post that here. The tutorial I've been following summarises calculating the errors and the gradient descent algorithm with the following equations:

For each training example $x$, compute the output error $\delta^{x, L}$ where $L =$ Final layer (Layer 1 in this case). $\delta^{x, L} = \nabla_aC_x \circ \sigma'(z^{x, L})$ where $\nabla_aC_x$ is the differential of the cost function (basic MSE) with respect to the Layer 1 activation output, and $\sigma'(z^{x, L})$ is the derivative of the sigmoid function of the Layer 1 output i.e. $\sigma(z^{x, L})(1-\sigma(z^{x, L}))$.

That's all good so far and I can calculate that quite straightforwardly. Now for $l = L-1, L-2, ...$, the error for each previous layer can be calculated as

$\delta^{x, l} = ((w^{l+1})^T \delta^{x, l+1}) \circ \sigma(z^{x, l})$

Which again, is pretty straight forward to implement.

Finally, to update the weights (and bias), the equations are for $l = L, L-1, ...$:

$w^l \rightarrow w^l - \frac{\eta}{m}\sum_x\delta^{x,l}(a^{x, l-1})^T$

$b^l \rightarrow b^l - \frac{\eta}{m}\sum_x\delta^{x,l}$

What I don't understand is how this works with vectors of different numbers of elements (I think the lack of vector notation here confuses me).

For example, Layer 1 has one neuron, so $\delta^{x, 1}$ will be a scalar value since it only outputs one value. However, $a^{x, 0}$ is a vector with two elements since layer 0 has two neurons. Which means that $\delta^{x, l}(a^{x, l-1})^T$ will be a vector even if I sum over all training samples $x$. What am I supposed to do here? Am I just supposed to sum the components of the vector as well?

Hopefully my question makes sense; I feel I'm very close to implementing this entirely and I'm just stuck here.

Thank you

[edit] Okay, so I realised that I've been misrepresenting the weights of the neurons and have corrected for that.

weights = [np.random.randn(y, x) for x, y in zip(sizes[:-1], sizes[1:])]

Which has the output

[array([[0.27660583, 1.00106314],
   [0.34017727, 0.74990392]])
array([[ 1.095244  , -0.22719165]])

Which means that layer0 has a weight matrix with shape 2x2 representing the 2 weights on neuron01 and the 2 weights on neuron02.

My understanding then is that $\delta^{x,l}$ has the same shape as the weights array because each weight gets updated indepedently. That's also fine.

But the bias term (according to the link I sourced) has 1 term for each neuron, which means layer 0 will has two bias terms (b00 and b01) and layer 1 has one bias term (b10).

However, to calculate the update for the bias terms, you sum the deltas over x i.e $\sum_x \delta^{x, l}$; if delta has the size of the weight matrix, then there are too many terms to update the bias terms. What have I missed here?

Many thanks


1 Answer 1


There seems to be a mismatch between the weights you provide and your network diagram. Since w[0] (the yellow connections) is meant to transform $ x \in \mathbb{R}^2 $ into the layer 0 activations which are $ \mathbb{R}^2 $, w[0] should be a matrix $ \in \mathbb{R}^{2 \times 2} $, not a vector in $\mathbb{R}^2 $ as you have. Likewise, your w[1] (the red connections) should be a vector $ \in \mathbb{R^2} $ and not a scalar. Finally, if you are indeed scaling the output of layer 1 (the blue connection), then you'll need an additional scalar value. However, the blue connection confuses me a bit as usually the activated output is used directly in the loss function, not a scaled version of it. Unless the blue connection stands for the loss function.

In short, I believe if you change the shapes of your weight matrices to actually represent your network diagram, your update equations will work. I'll go through the network below to make sure I illustrate my point.

$ x \in \mathbb{R}^{2} $, an input example

$ W^0 \in \mathbb{R}^{2 \times 2} $, the yellow connections

$ W^1 \in \mathbb{R}^2 $, the red connections

$ z^0 = xW^0 \in \mathbb{R}^{2} $, the weighted inputs to the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.

$ a^0 = \sigma(z^0) \in \mathbb{R}^{2} $, the output of the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.

$ z^1 = a^0 W^1 \in \mathbb{R} $, the weighted inputs to the layer 1 nodes. The dimensions of this should match the number of nodes at layer 1.

$ a^1 = \sigma(z^1) \in \mathbb{R} $, the output of the layer 1 nodes and thus the output of the network. The dimensions of this should match the number of nodes at layer 1.

Weight Updates

As you say before your edit, $\delta^1$, as the product of two scalars $\nabla_a C$ and $\sigma'(z^1)$, is also a scalar. Since $a^0$ is a vector in $\mathbb{R}^2$, then $\delta^1(a^0)^T$ is also a vector in $\mathbb{R}^2$. This matches what we expect, as it should match the dimensions of $W^1$ to allow the element-wise subtraction in the weight update equation.

NB. It is not the case, as you say in your edit, that the shape of $\delta^l$ should match the shape of $W^l$. It should instead match the number of nodes, and it is the shape of $\delta^l(a^{l-1})^T$ that should match the shape of $W^l$. You had this right in your original post.

Bias Updates

This brings us to the bias updates. There should be one bias term per node in a given layer, so the shapes of your biases are correct (i.e. $\mathbb{R}^2$ for layer 0 and $\mathbb{R}$ for layer 1). Now, we saw above that the shape of $\delta^l$ also matches the number of nodes in layer $l$, so again the element-wise subtraction in your original bias update equation works.

I also tried using this book to learn backprop, but I had a hard time connecting the variables with the different parts of the network and the corresponding code. I finally understood the algorithm in depth only after deriving all the update equations by hand for a very small network (2 inputs, one output, no hidden layers) and working my way up to larger networks, making sure to keep track of the shapes of the inputs and outputs along the way. If you're having trouble with the update equations I highly recommend this.

A final piece of advice that helped me: drop the $x$ and the summations over input examples from your formulations and just treat everything as matrices (e.g. a scalar becomes a matrix in $\mathbb{R}^{1 \times 1}$, $X$ is a matrix in $\mathbb{R}^{N \times D}$). First, this allows you to better interpret matrix orientations and debug issues such as a missing transpose operation. Second, this is (in my limited understanding) how backprop should actually be implemented in order to take advantage of optimized linalg libraries and GPUs, so it's perhaps a bit more relevant.

  • $\begingroup$ Thank you for the comment. I actually realised this mistake too and updated the weight variables to as you said. So for layer0, a 2x2 matrix is used to represent the weights of that layer and for layer1, a 2x1 matrix is used for the weights. This has solved the issue with updating the weights as far as I can tell (delta is a vector with the same shape as the weight vector now). The issue I have is with updating the biases now. If I understand correctly, each neuron should have only 1 bias term (layer0 has a 2x1 vector of biases, layer1 has a 1x1 vector of biases). $\endgroup$ Aug 12, 2020 at 11:32
  • $\begingroup$ But if delta has the shape of the weights, then how are the bias terms updated? I'll update my question to ask this in more detail. Thank you for the response. $\endgroup$ Aug 12, 2020 at 11:33
  • $\begingroup$ delta does not have the shape of the weights. I've edited my answer to clarify this and to include an explanation of bias updates. $\endgroup$ Aug 12, 2020 at 15:03
  • $\begingroup$ THank you for the reply again. Sorry, so for the NN I've specified, $W = [\R^{2x2}, \R^2]$ and $B = [\R^2, \R^1]$. Then $\sum_x \delta^{x, \mathbf{L}} (a^{x, \mathbf{L-1}})^T = [\R^{2x2}, \R^2]$ and $\sum_x \delta^{x, \mathbf{L}} = [\R^2, \R^1]$ Have I understood that correctly? Because you need as many update terms as you have weights and biases? $\endgroup$ Aug 12, 2020 at 17:48
  • $\begingroup$ Yes, exactly. The weight and bias updates are element-wise subtractions, so the updates need to have the same shape as the thing they're updating. Again, I highly recommend deriving the update equations for your network by applying the chain rule so you can check the shapes at each step. You're very welcome and best of luck with your implementation. Also, please accept my answer if you're satisfied. $\endgroup$ Aug 12, 2020 at 18:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .