Why do we lose detail of an image as we go deeper into a ConvNet?

Question

I was reading this research paper titled 'Image Style Transfer using Convolutional Neural Networks' which as the title suggests was based on Neural Style Transfer. I came across this line which didn't make immediate sense to me.

Here's how it went -

We reconstruct the input image from from layers ‘conv1 2’ (a), ‘conv2 2’ (b), ‘conv3 2’ (c), ‘conv4 2’ (d) and ‘conv5 2’ (e) of the original VGG-Network. We find that reconstruction from lower layers is almost perfect (a–c). In higher layers of the network, detailed pixel information is lost while the high-level content of the image is preserved (d,e).

The line that is italicised; Why does that happen?

Answer 1

The point of a convnet, or many kinds of neural networks in general, is to go from a lot of data down to a small piece of data. In classification tasks, for example, the input is all the pixels that make up a picture of a house, and the output is just the word "house" (or rather a number representing the word "house").

Obviously this process loses information. If you go from the word "house", back to a picture (i.e. you tell it "draw a house") you're probably going to get a completely different house!

In the style transfer task we have many numbers to describe the house picture with, not just the word "house", but we still have less than the full pixel data. Imagine that the intermediate representation represents something like "yellow wooden house with three windows and one window above and a red brick basement with 4 windows and the house is drawn at a 30 degree angle and there's a pink house to the left with two small windows below and one window above and a red roof is visible behind and above the pink house and ....."

If you had that representation, you could try to draw the original picture again, and the more information you have, the more accurately you can draw it. Early layers of the convnet contain information like "there's a vertical line at pixel coordinates 123,456" and later layers contain information like "there's a yellow house at pixel coordinates 123,456", and if you get to dense layers, they may just say "there's a yellow house".

Answer 2

A convolution layer has a Kernel that is matrix smaller that the image (in many papers 3*3), so this Kernel have learnable values.

The kernel is applied pixel by pixel (and dimensions, for color images), by applied I mean take the kernel centered in the pixel and multiply element-wise with the image and sum all results. This is the output for that pixel, a value that takes information from pixel's neighborhood.

When you apply this many times you lose specific pixel's information, but it gains information about pixel's neighborhood. And this is good because maybe this neighborhood is a corner, a line, curve, etc. (this is what learned the kernel to identify value content, for that reason the network in d and e have more basic forms and color without detail) . if you continue applying more convolutions you will find a abstract representation of the all the input, even you can get only one number that represent that input (if you don't avoid dimension reduction with padding or other tool) and say something like "oh it's a five and therefore the input is a house" (in classification case).

NOTE: This is very general and many details were omitted, if you want to learn more I recommend the book deep learning (Goodfellow et al., 2016) chapter 9 convolutional networks.