In a Convolutional Neural Network, unlike the fully connected layers, the same filter is used multiple times on the input while convolving - so during backpropagation, we get multiple derivatives for the filter parameters w.r.t the loss function. My question is, why do we sum all the derivatives to get the final gradient? Because, we don't sum the output of the convolution during forward pass. So, isn't it more sensible to average them? What is the intuition behind this?

PS: although I said CNN, what I'm actually doing is correlation for simplicity of learning.

  • $\begingroup$ To provide more context, can you provide the link to an implementation (or an explanation) where this is done (or stated)? Typically, you apply each filter only once to the image. You may have different filters that you apply to the image, but that's different than applying the exact same filter to the image. $\endgroup$ – nbro Nov 7 '20 at 11:20
  • 1
    $\begingroup$ X-posted: math.stackexchange.com/q/3897423 $\endgroup$ – Rodrigo de Azevedo Dec 20 '20 at 16:05

I really liked the question. Yes, we sum over derivatives. First of all think what backpropagation is trying to do: finding the affect of each parameter on the loss.

So as you said:

the same filter is used multiple times on the input while convolving

meaning that each kernel affects the final loss in several ways, so those affects should be summed together, not averaged.

  • $\begingroup$ Can you provide a link to a reliable implementation where the derivatives are summed? Moreover, I don't think this answer really clarifies why we need to sum, because you say "meaning that each kernel affects the final loss in several ways". Well, "several ways" does not imply that you need to sum the gradients. In any case, it's not necessarily true that each filter is applied to the image multiple times. It's usually applied once. What you actually do is to apply multiple different filters to the image, but they are usually different. So, can you clarify how you interpreted the question? $\endgroup$ – nbro Dec 4 '20 at 12:40
  • $\begingroup$ I don't know any reliable link to the implementation, but I've coded the backprop of conv layer myself (with numpy) and got the same result by doing so in tensorflow (using tf.gradients). $\endgroup$ – amin Dec 4 '20 at 22:08
  • $\begingroup$ When you apply a kernel to its input, you multiply the same kernel in different parts of input to produce the output of different neurons. So the same kernel affects the final loss in several ways (via different neurons). I think this gif helps to understand what I want to say. $\endgroup$ – amin Dec 4 '20 at 22:15
  • $\begingroup$ Ha, yes, you apply the kernel to different parts of the image: that's just the definition of the convolution/cross-correlation. Ok, now I understand what you mean when the OP said "apply the filter multiple times". $\endgroup$ – nbro Dec 4 '20 at 22:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.