1
$\begingroup$

Studying CNN Back-propagation I can't understand how can we compute the gradient of max pooling with overlapping regions ?

That's also a question from this quiz and can be also found on this book .

$\endgroup$
2
$\begingroup$

When gradients in a neural network can follow multiple paths to same parameter, the different gradient values from the sources can often be added together, because the operations in the forward direction are also sums and $\frac{d}{dx}(y+z) = \frac{dy}{dx} + \frac{dz}{dx}$.

That is the case already with gradients of kernels (which are sums over the image area), and is equally the case for overlapping aggregation, including maximums, minimums or averages.

So in the 1d case, if you have a max pool over the input params $[a_0, a_1, a_2, a_3, a_4]$ a max function $m_0 = max(a_0, a_1, a_2)$, $m_1 = max(a_2, a_3, a_4)$ which overlap at $a_2$, and gradients $\nabla_{\mathbf{m}} J = [\frac{\partial J}{\partial m_0}, \frac{\partial J}{\partial m_1}]$, then you would allocate those gradients to vector $\mathbf{a}$ according to which items in each group was the max of that group, adding them when they overlapped.

Examples:

If $\mathbf{a} = [3,0,1,2,0]$ and $\nabla_{\mathbf{m}}J = [0.7, 0.9]$, then $\nabla_{\mathbf{a}}J = [0.7, 0, 0, 0.9, 0]$

If $\mathbf{a} = [3,0,4,2,0]$ and $\nabla_{\mathbf{m}}J = [0.7, 0.9]$, then $\nabla_{\mathbf{a}}J = [0, 0, 1.6, 0, 0]$

| improve this answer | |
$\endgroup$
  • $\begingroup$ i got this answer , could you tell me if it logic ? denote x(h,w) input to max-pooling, and y(h,w) - output. Then dL/dx(h,w) = sum dL/dy(h',w') | over all y(h',w') which have been obtained from x(h.w) such that y(h',w') = x(h,w) $\endgroup$ – estamos Dec 15 '19 at 20:10
  • $\begingroup$ @estamos Sorry I don't understand your comment. Perhaps you could ask it as a new question on the site? $\endgroup$ – Neil Slater Dec 15 '19 at 21:17
  • $\begingroup$ I asked one professor this exact question and he responded this as an answer , I do not get it though . $\endgroup$ – estamos Dec 15 '19 at 21:52
  • 1
    $\begingroup$ @NeilSlater hi Neil, I tried editing your response to remove the orphan ")" at the end of your first sentence. I was denied because the edit was "too short". $\endgroup$ – user1269942 Dec 15 '19 at 21:54
  • $\begingroup$ @user1269942: Thanks for pointing outthe typo. Fixed now $\endgroup$ – Neil Slater Dec 16 '19 at 7:14
0
$\begingroup$

Denote x(h,w) input to max-pooling, and y(h,w) - output.

Then $\frac{dL}{dx}(h,w)= \sum \frac{dL}{dy}(h',w')$

over all y(h',w') which have been obtained from x(h.w) such that y(h',w') = x(h,w).

related to this p 11

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ So this is the same answer as mine (add the gradients when they overlap), but for the 2D case. $\endgroup$ – Neil Slater Dec 16 '19 at 7:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.