# Implementing Gradient Descent Algorithm in Python, bit confused regarding equations

I'm following the guide as outlined at this link: http://neuralnetworksanddeeplearning.com/chap2.html

For the purposes of this question, I've written a basic network 2 hidden layers, one with 2 neurons and one with one neuron. For a very basic task, the network will learn how to compute an OR logic gate so the training data will be:

X = [[0, 0], [0, 1], [1, 0], [1, 1]]
Y = [0, 1, 1, 1]


For this example, the weights and biases are:

w = [[0.3, 0.4], [0.1]]
b = [[1, 1], ]


The feedforward part was pretty easy to implement so I don't think I need to post that here. The tutorial I've been following summarises calculating the errors and the gradient descent algorithm with the following equations:

For each training example $$x$$, compute the output error $$\delta^{x, L}$$ where $$L =$$ Final layer (Layer 1 in this case). $$\delta^{x, L} = \nabla_aC_x \circ \sigma'(z^{x, L})$$ where $$\nabla_aC_x$$ is the differential of the cost function (basic MSE) with respect to the Layer 1 activation output, and $$\sigma'(z^{x, L})$$ is the derivative of the sigmoid function of the Layer 1 output i.e. $$\sigma(z^{x, L})(1-\sigma(z^{x, L}))$$.

That's all good so far and I can calculate that quite straightforwardly. Now for $$l = L-1, L-2, ...$$, the error for each previous layer can be calculated as

$$\delta^{x, l} = ((w^{l+1})^T \delta^{x, l+1}) \circ \sigma(z^{x, l})$$

Which again, is pretty straight forward to implement.

Finally, to update the weights (and bias), the equations are for $$l = L, L-1, ...$$:

$$w^l \rightarrow w^l - \frac{\eta}{m}\sum_x\delta^{x,l}(a^{x, l-1})^T$$

$$b^l \rightarrow b^l - \frac{\eta}{m}\sum_x\delta^{x,l}$$

What I don't understand is how this works with vectors of different numbers of elements (I think the lack of vector notation here confuses me).

For example, Layer 1 has one neuron, so $$\delta^{x, 1}$$ will be a scalar value since it only outputs one value. However, $$a^{x, 0}$$ is a vector with two elements since layer 0 has two neurons. Which means that $$\delta^{x, l}(a^{x, l-1})^T$$ will be a vector even if I sum over all training samples $$x$$. What am I supposed to do here? Am I just supposed to sum the components of the vector as well?

Hopefully my question makes sense; I feel I'm very close to implementing this entirely and I'm just stuck here.

Thank you

 Okay, so I realised that I've been misrepresenting the weights of the neurons and have corrected for that.

weights = [np.random.randn(y, x) for x, y in zip(sizes[:-1], sizes[1:])]


Which has the output

[array([[0.27660583, 1.00106314],
[0.34017727, 0.74990392]])
array([[ 1.095244  , -0.22719165]])


Which means that layer0 has a weight matrix with shape 2x2 representing the 2 weights on neuron01 and the 2 weights on neuron02.

My understanding then is that $$\delta^{x,l}$$ has the same shape as the weights array because each weight gets updated indepedently. That's also fine.

But the bias term (according to the link I sourced) has 1 term for each neuron, which means layer 0 will has two bias terms (b00 and b01) and layer 1 has one bias term (b10).

However, to calculate the update for the bias terms, you sum the deltas over x i.e $$\sum_x \delta^{x, l}$$; if delta has the size of the weight matrix, then there are too many terms to update the bias terms. What have I missed here?

Many thanks

There seems to be a mismatch between the weights you provide and your network diagram. Since w (the yellow connections) is meant to transform $$x \in \mathbb{R}^2$$ into the layer 0 activations which are $$\mathbb{R}^2$$, w should be a matrix $$\in \mathbb{R}^{2 \times 2}$$, not a vector in $$\mathbb{R}^2$$ as you have. Likewise, your w (the red connections) should be a vector $$\in \mathbb{R^2}$$ and not a scalar. Finally, if you are indeed scaling the output of layer 1 (the blue connection), then you'll need an additional scalar value. However, the blue connection confuses me a bit as usually the activated output is used directly in the loss function, not a scaled version of it. Unless the blue connection stands for the loss function.

In short, I believe if you change the shapes of your weight matrices to actually represent your network diagram, your update equations will work. I'll go through the network below to make sure I illustrate my point.

$$x \in \mathbb{R}^{2}$$, an input example

$$W^0 \in \mathbb{R}^{2 \times 2}$$, the yellow connections

$$W^1 \in \mathbb{R}^2$$, the red connections

$$z^0 = xW^0 \in \mathbb{R}^{2}$$, the weighted inputs to the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.

$$a^0 = \sigma(z^0) \in \mathbb{R}^{2}$$, the output of the layer 0 nodes. The dimensions of this should match the number of nodes at layer 0.

$$z^1 = a^0 W^1 \in \mathbb{R}$$, the weighted inputs to the layer 1 nodes. The dimensions of this should match the number of nodes at layer 1.

$$a^1 = \sigma(z^1) \in \mathbb{R}$$, the output of the layer 1 nodes and thus the output of the network. The dimensions of this should match the number of nodes at layer 1.

As you say before your edit, $$\delta^1$$, as the product of two scalars $$\nabla_a C$$ and $$\sigma'(z^1)$$, is also a scalar. Since $$a^0$$ is a vector in $$\mathbb{R}^2$$, then $$\delta^1(a^0)^T$$ is also a vector in $$\mathbb{R}^2$$. This matches what we expect, as it should match the dimensions of $$W^1$$ to allow the element-wise subtraction in the weight update equation.

NB. It is not the case, as you say in your edit, that the shape of $$\delta^l$$ should match the shape of $$W^l$$. It should instead match the number of nodes, and it is the shape of $$\delta^l(a^{l-1})^T$$ that should match the shape of $$W^l$$. You had this right in your original post.

This brings us to the bias updates. There should be one bias term per node in a given layer, so the shapes of your biases are correct (i.e. $$\mathbb{R}^2$$ for layer 0 and $$\mathbb{R}$$ for layer 1). Now, we saw above that the shape of $$\delta^l$$ also matches the number of nodes in layer $$l$$, so again the element-wise subtraction in your original bias update equation works.
A final piece of advice that helped me: drop the $$x$$ and the summations over input examples from your formulations and just treat everything as matrices (e.g. a scalar becomes a matrix in $$\mathbb{R}^{1 \times 1}$$, $$X$$ is a matrix in $$\mathbb{R}^{N \times D}$$). First, this allows you to better interpret matrix orientations and debug issues such as a missing transpose operation. Second, this is (in my limited understanding) how backprop should actually be implemented in order to take advantage of optimized linalg libraries and GPUs, so it's perhaps a bit more relevant.
• THank you for the reply again. Sorry, so for the NN I've specified, $W = [\R^{2x2}, \R^2]$ and $B = [\R^2, \R^1]$. Then $\sum_x \delta^{x, \mathbf{L}} (a^{x, \mathbf{L-1}})^T = [\R^{2x2}, \R^2]$ and $\sum_x \delta^{x, \mathbf{L}} = [\R^2, \R^1]$ Have I understood that correctly? Because you need as many update terms as you have weights and biases? – user1147964 Aug 12 '20 at 17:48