5
$\begingroup$

I read that to compute the derivative of the error with respect to the input of a convolution layer is the same to make of a convolution between deltas of the next layer and the weight matrix rotated by $180°$, i.e. something like

$$\delta^l_{ij}=\delta^{l+1}_{ij} * rot180(W^{l+1})f'(x^l_{ij})$$

with $*$ convolution operator; This is valid with $stride=1$; but what happens when stride is greater than $1$? is still a convolution with a kernel rotation or I can't make this simplification?

$\endgroup$
  • 1
    $\begingroup$ Did you miss something with the formatting? $\endgroup$ – DuttaA Mar 27 '18 at 8:57
2
$\begingroup$

Animation of backpropagation with stride > 1

Backpropagation with stride > 1 involves dilation of the gradient tensor with stride-1 zeroes. I created a blog post that describes this in greater detail.

$\endgroup$
1
+25
$\begingroup$

From the paper found from the post linked below:

'We find that max-pooling can simply be replaced by a convolutional layer with increased stride without loss in accuracy on several image recognition benchmarks'

All that means that only values are skipped (=pooling is made) to the matrix, otherwise all works like a convolution should do.

Sources:

https://arxiv.org/pdf/1412.6806.pdf

https://stackoverflow.com/questions/44666390/max-pool-layer-vs-convolution-with-stride-performance

$\endgroup$
  • $\begingroup$ This is not an answer to my question; you are talking about pooling, my question is about simple convolution in backpropagation algorithm. $\endgroup$ – volperossa Apr 2 '18 at 14:52
  • $\begingroup$ Well, the point is that strides introduce pooling kind of phenomenom and otherwise it does not change CNN performance and if I read my source right, also correctness. $\endgroup$ – mico Apr 2 '18 at 14:55
  • $\begingroup$ Probably more direct professional expertice you could get on datascience.SE site. I belong to both of these, thus I knew sth of the issue. $\endgroup$ – mico Apr 2 '18 at 15:01
1
$\begingroup$

I have just the same problem, and I was trying to derive the backpropagation for the conv layer with stride, but it doesn't work. When you do the striding in forward propagation, you chose the elements next to each other to convolve with the kernel, than take a step >1. This results in the fact that in the backprop, in the reverse operation the delta matrix elements will be multiplied by the kernel elements, (with the rotation) but not as strided, but you are picking elements that are not next to each other, something like DY_11 * K_11 + DY_13 * K_12 + DY_31 * K_21 + DY_33 * K_22, which is NOT the equivalent as a convolution with a stride >1.

So as far as I am concerned, if I would like to implement the convnet by myself to get a better grasp of the concept, I have to implement a different method for the backprop, if I allow strides.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.