I am trying to implement CNN using python NumPy. I searched so much, but all I found was for one filter with one channel for convolution.

Suppose $x$ is an image with the shape: (N_Height, N_Width, N_Channel) = (5,5,3).

Let's say I have 16 filters with this shape: (F_Height, F_Width, N_Channel) = (3,3,3) , stride=1 and padding=0


The output shape after convolution 2d will be

math.floor((N_Height - F_Height + 2*padding)/stride + 1 )),
math.floor((N_Width- F_Width + 2*padding)/stride + 1 )),

So, the output of this layer will be an array with this shape: (Height, Width, Channel) = (3, 3, 16)


Suppose $dL/dh$ is the input for my layer in back-propagation with this shape: (3, 3, 16)

Now, I must find $dL/dw$ and $dL/dx$: $dL/dw$ to update my filters parameter and $dL/dx$ to pass it as input to the previous layer as the loss respect to the input $x$.

From this answer Error respect to filters weights I found how to calculate $dL/dw$.

The problem I have in the back-propagation is I don't know how to calculate $dL/dx$ having this shape: (5, 5, 3) and pass it to the previous layer.

I read lots of articles in Medium and other sites, but I don't get how to calculate it:


1 Answer 1


While this may not be the answer you were looking for, I hope this explanation will help you to understand applying backpropagation to a CNN. Fundamentally, convolutional layers are no different than dense layers, however there are restrictions. The key one is weight-sharing which allows a CNN to be much more efficient than a regular dense layer (as well as it being sparse due to locality). Imagine we are transforming a 4x4 image into a 2x2 image. Since we are inputting a 16-vector, and outputting a 4-vector, we need a weights matrix of 4x16:

enter image description here

This has 64 parameters. In a convolutional layer, we can accomplish this by convolving a 3x3 kernel over the image:

$$ K= \begin{bmatrix} k_{1,1} & k_{1,2} & k_{1,3} \\ k_{2,1} & k_{2,2} & k_{2,3} \\ k_{3,1} & k_{3,2} & k_{3,3} \end{bmatrix} $$

This convolution is equivalent to multiplying by the weights matrix:

enter image description here

As you can see, this only requires 9 parameters and backpropagation can be applied to update these parameters.

Image Source: https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1

  • $\begingroup$ thanks but as you said it is not my question's answer. I finally find $dL/dX$ for each channel in my example 16 then I add all of them and I get my $dL/dX$ $\endgroup$
    – A.Najafi
    May 27, 2020 at 8:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .