Is it possible to reconstruct convolutional layers' input using transposed convolution?

Question

I've been trying to visualize internal activations in CNN and came across this paper: "Visualizing and Understanding Convolutional Networks" by Zeiler & Fergus.

In the paper they mentioned reconstructing the input image from internal convnet activations using deconvnet. Specifically, on reversing the convolutional (filter) layers, they said:

To invert this, the deconvnet uses transposed versions of the same filters, but applied to the rectified maps, not the output of the layer beneath. In practice this means flipping each filter vertically and horizontally.

I believe they are referring to just a simple transposed convolution operation, since convolution with flipped weights is equivalent to applying the transposed convolution operation.

My question is that transposed convolution is not an inverse of the convolution operation. This simple snippet shows just that:

import torch
import torch.nn as nn
i = torch.randn((10,10)).unsqueeze(0)
c = nn.Conv2d(1, 1, 2, bias=False)
ct = nn.ConvTranspose2d(1, 1, 2, bias=False)
ct.weight = nn.Parameter(c.weight)

torch.isclose(i, ct(c(i)) # not true

So I don't really understand how they claim that the output from deconvnet is a representation of the internal activations of the convnet.

Answer 1

You're not able reconstruct convolutional layers' inputs using transposed convolutions (in most cases). The term invert is a bit confusing here -- I interpret this to mean inverting the space of inputs and outputs, not the values themselves. If you look at section 2.1, for example, they state: "We present a novel way to map these activities back to the input pixel space, showing what input pattern originally caused a given activation in the feature maps."

In your code snippet, even though the values of i and ct(c(i) are different, the shape should be the same, as ct transforms the activations from post-convolutional space to pre-convolutional space.

You can see this in your snippet, but also in the mathematical formulation of convolutions and transposed convolutions. Let $W \in \mathbb{R}^{(n,m)}$ be the sparse matrix representation of the convolutional kernel, then, as you mentioned, $W^T \in \mathbb{R}^{(m,n)}$ is the transposed convolutional kernel (see the previous link for specifics on how this works).

For some input $x \in \mathbb{R}^n$, if you do $W^T W x$, you get another $\mathbb{R}^n$ vector back but $W^T W \neq I$ in most cases (i.e., unless $W$ is orthogonal).

Now, let's think about this in the context of this paper (i.e., what's the point of these deconvolution operations?). A CNN can be thought of as a feature extractor which converts the high-dimensional input representation into a low(er) dimensional, dense representation that contains the most important features in the image for the classification task such that you should be able to linearly separate the target classes in this dense, lower dimensional feature space. Because this feature space is of much lower dimension than the input image, you can't fit all of the information in the input image into this vector. So, each layer should be iteratively extracting important features from their inputs. By design, the convolutional layers aren't invertible -- they're selecting the important features and throwing away the unimportant ones.

Because of this, if you reverse the convolution operations, yes, the image now looks different, but this difference tells you what the convolution operation is doing!

Answer 2

Alexander has some great explanations above.

After doing some more research myself I came up with some understandings as well.

One interpretation of transposed convolution is that it can be seen as computing gradients w.r.t. input, for a convolution operation. E.g.

import torch
import torch.nn as nn

i = torch.randn((5,5))
c = nn.Conv2d(1,1,2,bias=False)
ct = nn.ConvTranspose2d(1,1,2,bias=False)
ct.weight = nn.Parameter(c.weight)

o = c(i)
o.retain_grad()
o.sum().backward()

torch.isclose(i.grad, ct(o.grad())) # this is true

Thus the deconvnet's output can be roughly interpreted as computing scaled gradients w.r.t. the input. This certainly gives some insights into what the internal activations represent, since by definition, pixels with bigger (absolute) gradients have the most impact on the output of the network / selected neuron.