Skip to main content
2 of 2
added 16 characters in body
Alexander Wan
  • 1.4k
  • 1
  • 12

You're not able reconstruct convolutional layers' inputs using transposed convolutions (in most cases). The term invert is a bit confusing here -- I interpret this to mean inverting the space of inputs and outputs, not the values themselves. If you look at section 2.1, for example, they state: "We present a novel way to map these activities back to the input pixel space, showing what input pattern originally caused a given activation in the feature maps."

In your code snippet, even though the values of i and ct(c(i) are different, the shape should be the same, as ct transforms the activations from post-convolutional space to pre-convolutional space.

You can see this in your snippet, but also in the mathematical formulation of convolutions and transposed convolutions. Let $W \in \mathbb{R}^{(n,m)}$ be the sparse matrix representation of the convolutional kernel, then, as you mentioned, $W^T \in \mathbb{R}^{(m,n)}$ is the transposed convolutional kernel (see the previous link for specifics on how this works).

For some input $x \in \mathbb{R}^n$, if you do $W^T W x$, you get another $\mathbb{R}^n$ vector back but $W^T W \neq I$ in most cases (i.e., unless $W$ is orthogonal).

Now, let's think about this in the context of this paper (i.e., what's the point of these deconvolution operations?). A CNN can be thought of as a feature extractor which converts the high-dimensional input representation into a low(er) dimensional, dense representation that contains the most important features in the image for the classification task such that you should be able to linearly separate the target classes in this dense, lower dimensional feature space. Because this feature space is of much lower dimension than the input image, you can't fit all of the information in the input image into this vector. So, each layer should be iteratively extracting important features from their inputs. By design, the convolutional layers aren't invertible -- they're selecting the important features and throwing away the unimportant ones.

Because of this, if you reverse the convolution operations, yes, the image now looks different, but this difference tells you what the convolution operation is doing!

Alexander Wan
  • 1.4k
  • 1
  • 12