0
$\begingroup$

In a Convolutional Neural Network, unlike the fully connected layers, the same filter is used multiple times on the input while convolving - so during backpropagation, we get multiple derivatives for the filter parameters w.r.t the loss function. My question is, why do we sum all the derivatives to get the final gradient? Because, we don't sum the output of the convolution during forward pass. So, isn't it more sensible to average them? What is the intuition behind this?

PS: although I said CNN, what I'm actually doing is correlation for simplicity of learning.

Improve this question
$\endgroup$
2
  • $\begingroup$ To provide more context, can you provide the link to an implementation (or an explanation) where this is done (or stated)? Typically, you apply each filter only once to the image. You may have different filters that you apply to the image, but that's different than applying the exact same filter to the image. $\endgroup$
    – nbro
    Nov 7, 2020 at 11:20
  • 1
    $\begingroup$ X-posted: math.stackexchange.com/q/3897423 $\endgroup$
    – Rodrigo de Azevedo
    Dec 20, 2020 at 16:05

1 Answer 1

Reset to default
0
$\begingroup$

I really liked the question. Yes, we sum over derivatives. First of all think what backpropagation is trying to do: finding the affect of each parameter on the loss.

So as you said:

the same filter is used multiple times on the input while convolving

meaning that each kernel affects the final loss in several ways, so those affects should be summed together, not averaged.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Can you provide a link to a reliable implementation where the derivatives are summed? Moreover, I don't think this answer really clarifies why we need to sum, because you say "meaning that each kernel affects the final loss in several ways". Well, "several ways" does not imply that you need to sum the gradients. In any case, it's not necessarily true that each filter is applied to the image multiple times. It's usually applied once. What you actually do is to apply multiple different filters to the image, but they are usually different. So, can you clarify how you interpreted the question? $\endgroup$
    – nbro
    Dec 4, 2020 at 12:40
  • $\begingroup$ I don't know any reliable link to the implementation, but I've coded the backprop of conv layer myself (with numpy) and got the same result by doing so in tensorflow (using tf.gradients). $\endgroup$
    – amin
    Dec 4, 2020 at 22:08
  • $\begingroup$ When you apply a kernel to its input, you multiply the same kernel in different parts of input to produce the output of different neurons. So the same kernel affects the final loss in several ways (via different neurons). I think this gif helps to understand what I want to say. $\endgroup$
    – amin
    Dec 4, 2020 at 22:15
  • $\begingroup$ Ha, yes, you apply the kernel to different parts of the image: that's just the definition of the convolution/cross-correlation. Ok, now I understand what you mean when the OP said "apply the filter multiple times". $\endgroup$
    – nbro
    Dec 4, 2020 at 22:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .