1
$\begingroup$

Why does DeconvNet (Zeiler, 2014) use ReLU in the backward pass (after unpooling)? Are not the feature maps values already positive due to the ReLU in the forward pass? So, why do the authors apply the ReLU again coming back to the input?

update: I better explain my problem:

given an input image $x$ and ConvLayer $CL$ composed of:

  1. a convolution
  2. an activation function ReLU
  3. a pool operation

$f$ is the output of ConvLayer given an input $x$, i.e. $f=CL(x)$.

So, the Deconv target is to "reverse" the output $f$ (the feature map) to restore an approximate version of $x$. To this aim, the authors define a function $CL^{-1}$ composed of 3 subfunctions:

a. unpool

b. activation function ReLU (useless in my opinion, because $f$ is already positive due to the application of the 2. step in $CL(f)$)

c. transposed convolution.

In other words $x\simeq CL^{-1}(f)$ where $CL^{-1} (f) = transpconv(relu(unpool(f)))$. But, if $f$ is the output computed as $f=CL(x)$, it is already positive, so the b. step is useless.

This is what I understood from the paper. Where I wrong?

$\endgroup$
  • $\begingroup$ While computing error (training phase) its just like any other layer. Error can be negative, hence the ReLU, I guess. $\endgroup$ – Karan Shah Apr 3 at 10:15
  • $\begingroup$ but in the paper seems that $CL^{-1}$ is applied to $f$ (which is the feature map computed in the forward pass) $\endgroup$ – volperossa Apr 11 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.