How do I calculate the partial derivative with respect to $x$?

Question

I am trying to implement CNN using python NumPy. I searched so much, but all I found was for one filter with one channel for convolution.

Suppose $x$ is an image with the shape: (N_Height, N_Width, N_Channel) = (5,5,3).

Let's say I have 16 filters with this shape: (F_Height, F_Width, N_Channel) = (3,3,3) , stride=1 and padding=0

Forward:

The output shape after convolution 2d will be

(
math.floor((N_Height - F_Height + 2*padding)/stride + 1 )),
math.floor((N_Width- F_Width + 2*padding)/stride + 1 )),
filter_count
)

So, the output of this layer will be an array with this shape: (Height, Width, Channel) = (3, 3, 16)

BackPropagation:

Suppose $dL/dh$ is the input for my layer in back-propagation with this shape: (3, 3, 16)

Now, I must find $dL/dw$ and $dL/dx$: $dL/dw$ to update my filters parameter and $dL/dx$ to pass it as input to the previous layer as the loss respect to the input $x$.

From this answer Error respect to filters weights I found how to calculate $dL/dw$.

The problem I have in the back-propagation is I don't know how to calculate $dL/dx$ having this shape: (5, 5, 3) and pass it to the previous layer.

I read lots of articles in Medium and other sites, but I don't get how to calculate it:

• How Backpropagation works in a CNN
• The best explanation of Convolutional Neural Networks on the Internet!
• Backpropagation In Convolutional Neural Networks
• How to propagate error back to previous layer in CNN?

Answer 1

While this may not be the answer you were looking for, I hope this explanation will help you to understand applying backpropagation to a CNN. Fundamentally, convolutional layers are no different than dense layers, however there are restrictions. The key one is weight-sharing which allows a CNN to be much more efficient than a regular dense layer (as well as it being sparse due to locality). Imagine we are transforming a 4x4 image into a 2x2 image. Since we are inputting a 16-vector, and outputting a 4-vector, we need a weights matrix of 4x16:

This has 64 parameters. In a convolutional layer, we can accomplish this by convolving a 3x3 kernel over the image:

$$ K= \begin{bmatrix} k_{1,1} & k_{1,2} & k_{1,3} \\ k_{2,1} & k_{2,2} & k_{2,3} \\ k_{3,1} & k_{3,2} & k_{3,3} \end{bmatrix} $$

This convolution is equivalent to multiplying by the weights matrix:

As you can see, this only requires 9 parameters and backpropagation can be applied to update these parameters.

Image Source: https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1