I've been trying to learn backpropagation for CNNs. I read several articles like this one and this one. They all say that to compute the gradients for the filters, you just do a convolution with the input volume as input and the error matrix as the kernel. After that, you just subtract the filter weights by the gradients(multiplied by the learning rate). I implemented this process but it's not working.

Here's a simple example that I tried:

Input volume (randomised)

 1 -1  0
 0  1  0
 0 -1  1

In this case, we want the filter to only pick up the top left 4 elements. So the target output will be:

 1  0(supposed to be -1, but ReLU is applied) 
 0  1

We know that the desired filter is:

 1  0
 0  0

But we pretend that we don't know this.

We first randomise a filter:

 1 -1
 1  1

The output right now is:

 3  0 
-2  1

Apply ReLU:

 3  0
 0  1

Error (target - output):

-2  0
 0  0

Use error as kernel to compute gradients:

-2  2
 0 -2

Say the learning rate is 0.5, then the new filter is:

2 -2
1  2

This is still wrong! It's not improving at all. If this process is repeated, it won't learn the desired filter. So I must have understood the math wrong. So what's the problem here?


1 Answer 1


You have made many wrong assumptions in this question. First theoretically speaking,

  • Filters do not work in the way to 'pick up elements' (they work on principle of edge detection).
  • You have assumed only a single combination of filter weights will give the desired output (assuming continuous weights not binary). This is especially in prominence in the problem of Regularization where we want to choose a set of weights without over-fitting data.
  • The error you used looks very similar to Perceptron update rule (squared error gives the same derivative, but make sure you are not confusing the two).
  • Backpropagation through 'dead ReLu's' is not possible (see this answer for more details).

Now, let us check mathematically:

Input Volume:

$$ \begin{matrix} 1 & -1 & 0 \\ 0 & 1 & 0 \\ 0 & -1 & 1 \\ \end{matrix} $$

Desired Output:

$$ \begin{matrix} 1 & -1 \\ 0 & 1 \\ \end{matrix} $$

Note, in this step you desire an output which is negative (element (0,-1)), but you are forward propagating through a ReLu which is cutting off the negative part, thus the gradients have no way to communicate or update the required negative. Basically,

$ wx \rightarrow ReLu \rightarrow y$ is happening and if 'x' is a negative number then $y$ is always $0$ thus $(target-y)$ is always $target$ and hence whatever the value of $x$ error remains constant, and if we want to backpropagate (assuming squared error) then:

$\frac{d}{dw} (target - y)^2 = 2*(target - y)*\frac{d}{dw}y = -2*(target - y)*0 = 0$ (Remember from the ReLu output graph slope is $0$ in the negative region).

Now, you randomise a filter:

$$ \begin{matrix} 1 & -1 \\ 1 & 1 \\ \end{matrix} $$

apply ReLu and get the following:

$$ \begin{matrix} 3 & 0 \\ 0 & 1 \\ \end{matrix} $$

again you have chosen you target to have a negative number which is not possible in the case of ReLu activation.

But continuing you get the error as:

$$ \begin{matrix} -2 & -1 \\ 0 & 0 \\ \end{matrix} $$

use error to compute gradients (which again you have calculated wrong,by Backpropagating through ReLu's with ) values and also missed the minus sign associated with the output, but you have compensated it by adding it to the $w$ whereas the convention is to subtract):

$$ \begin{matrix} 1 & 2 \\ 1 & -2 \\ \end{matrix} $$

And get the new filter as:

$$ \begin{matrix} 0.5 & 0 \\ 0.5 & 0 \\ \end{matrix} $$

This is a pretty good approximation of the desired filter (even though the previous steps have wrong assumptions, but it does not matter much, since what you essentially did was use a linear activation function which will work, if you go through enough iterations). So basically you are using a Linear Filters and the details are too hodge podge for me to go into, so I will suggest some resources for you to see ReLu backpropagation:

Deep Neural Network - Backpropogation with ReLU

  • $\begingroup$ NOTE: answer given in context of the older unedited question, newer question requires only change of calculation but the ideas remain same. $\endgroup$
    – user9947
    May 21, 2019 at 6:16
  • $\begingroup$ Thanks for your answer. I still have three more questions: 1. So my understanding for the gradient calculation is basically correct and I just need fix the problem in the target matrix and train it with more iterations, right? 2. The trained filter may not be what I expect but it will still get the job done, right? 3. If I were to use a different activation function (e.g. sigmoid), is the whole thing still the same? $\endgroup$ May 21, 2019 at 10:10
  • $\begingroup$ @YixuanWang 1.) I couldn't understand what problem you are talking about. 2.) Yes, it might have multiple filter solutions depending on the type of linear equation you are trying to solve I.e. just look at at it like a 4 variable linear equation 3.) By same if you mean after you made the necessary adjustments to your target after passing through sigmoid then yes $\endgroup$
    – user9947
    May 21, 2019 at 11:08
  • $\begingroup$ Also the interpretation that you select a part of picture is wrong, it is a general Target and the filter will try to achieve it without any kind of intelligent decision that 'hey if we choose this filter then we will achieve our goal' instead it is iteratively trying to achieve it and may come to very different results and maybe even a non optimal result depending on the training. $\endgroup$
    – user9947
    May 21, 2019 at 11:10
  • $\begingroup$ The problem I talked about was that I forgot to apply ReLU to the target matrix. Also I made it select part of the input matrix because I wanted to make it simpler. $\endgroup$ May 21, 2019 at 11:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .