I am trying to implement CNN using python Numpy.
I searched so much, but all I found was for one filter with one channel for Convolution.

Suppose we have an X as Image with this shape: (N_Height, N_Width, N_Channel) = (5,5,3)

And Let's say I have 16 filters with this shape: (F_Height, F_Width, N_Channel) = (3,3,3) , stride=1 and padding=0


Output shape after conv2d will be

math.floor((N_Height - F_Height + 2*padding)/stride + 1 )),
math.floor((N_Width- F_Width + 2*padding)/stride + 1 )),

So the output of this layer will be an array with this shape: (Height, Width, Channel) = (3, 3, 16)


Suppose $dL/dh$ is the input for my layer in backpropagation with this shape: (3,3,16)

Now I must find $dL/dw$ and $dL/dx$: $dL/dw$ to update my filters params and $dL/dx$ to pass it as input to the previous layer as Loss respect to the input X.

From this answer Error respect to filters weights I found how to calculate $dL/dw$.

The problem I have in BackPropagation is I don't know how to calculate $dL/dx$ having this shape:(5,5,3) and pass it to the prev layer.

I read lots of articles in Medium and other sites but I don't get how to calculate it:

How Backpropagation works in a CNN
The best explanation of Convolutional Neural Networks on the Internet!
Backpropagation In Convolutional Neural Networks
How to propagate error back to previous layer in CNN?

Thanks in advance :)

  • 1
    $\begingroup$ Hi and welcome to AI SE! Please, format your symbols of the derivates with MathJax, i.e. wrap them with two $ symbols. $\endgroup$ – nbro May 16 '20 at 13:11
  • $\begingroup$ Thanks, I did . $\endgroup$ – A.Najafi May 16 '20 at 13:26

While this may not be the answer you were looking for, I hope this explanation will help you to understand applying backpropagation to a CNN. Fundamentally, convolutional layers are no different than dense layers, however there are restrictions. The key one is weight-sharing which allows a CNN to be much more efficient than a regular dense layer (as well as it being sparse due to locality). Imagine we are transforming a 4x4 image into a 2x2 image. Since we are inputting a 16-vector, and outputting a 4-vector, we need a weights matrix of 4x16:

enter image description here

This has 64 parameters. In a convolutional layer, we can accomplish this by convolving a 3x3 kernel over the image:

$$ K= \begin{bmatrix} k_{1,1} & k_{1,2} & k_{1,3} \\ k_{2,1} & k_{2,2} & k_{2,3} \\ k_{3,1} & k_{3,2} & k_{3,3} \end{bmatrix} $$

This convolution is equivalent to multiplying by the weights matrix:

enter image description here

As you can see, this only requires 9 parameters and backpropagation can be applied to update these parameters.

Image Source: https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1

  • $\begingroup$ thanks but as you said it is not my question's answer. I finally find $dL/dX$ for each channel in my example 16 then I add all of them and I get my $dL/dX$ $\endgroup$ – A.Najafi May 27 '20 at 8:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.