0
$\begingroup$

I was reading this research paper titled 'Image Style Transfer using Convolutional Neural Networks' which as the title suggests was based on Neural Style Transfer. I came across this line which didn't make immediate sense to me.

enter image description here

Here's how it went -

We reconstruct the input image from from layers ‘conv1 2’ (a), ‘conv2 2’ (b), ‘conv3 2’ (c), ‘conv4 2’ (d) and ‘conv5 2’ (e) of the original VGG-Network. We find that reconstruction from lower layers is almost perfect (a–c). In higher layers of the network, detailed pixel information is lost while the high-level content of the image is preserved (d,e).

The line that is italicised; Why does that happen?

$\endgroup$
6
  • 1
    $\begingroup$ The excerpt you're quoting is not saying that you lose detail of an image as you go deeper, but that using a certain output of a certain layer rather than another leads to "better" reconstructed images. So, you may want to edit your post to clarify that. $\endgroup$
    – nbro
    Aug 6 at 22:54
  • $\begingroup$ It says "detailed pixel information is lost". How does that mean you have better images as you go deeper in the network ? $\endgroup$
    – HarshDarji
    Aug 7 at 6:15
  • 1
    $\begingroup$ Yes, but because you're trying to reconstruct the pixels of the images, as far as I understand. $\endgroup$
    – nbro
    Aug 8 at 21:29
  • $\begingroup$ Ohh okay. But at the end of the CNN are the pixels reconstructed better right? $\endgroup$
    – HarshDarji
    Aug 9 at 6:30
  • 1
    $\begingroup$ Does this answers your question? stackoverflow.com/questions/28953622/… $\endgroup$
    – sam2611
    Sep 5 at 13:40
1
$\begingroup$

The point of a convnet, or many kinds of neural networks in general, is to go from a lot of data down to a small piece of data. In classification tasks, for example, the input is all the pixels that make up a picture of a house, and the output is just the word "house" (or rather a number representing the word "house").

Obviously this process loses information. If you go from the word "house", back to a picture (i.e. you tell it "draw a house") you're probably going to get a completely different house!

In the style transfer task we have many numbers to describe the house picture with, not just the word "house", but we still have less than the full pixel data. Imagine that the intermediate representation represents something like "yellow wooden house with three windows and one window above and a red brick basement with 4 windows and the house is drawn at a 30 degree angle and there's a pink house to the left with two small windows below and one window above and a red roof is visible behind and above the pink house and ....."

If you had that representation, you could try to draw the original picture again, and the more information you have, the more accurately you can draw it. Early layers of the convnet contain information like "there's a vertical line at pixel coordinates 123,456" and later layers contain information like "there's a yellow house at pixel coordinates 123,456", and if you get to dense layers, they may just say "there's a yellow house".

$\endgroup$
1
$\begingroup$

A convolution layer has a Kernel that is matrix smaller that the image (in many papers 3*3), so this Kernel have learnable values.

The kernel is applied pixel by pixel (and dimensions, for color images), by applied I mean take the kernel centered in the pixel and multiply element-wise with the image and sum all results. This is the output for that pixel, a value that takes information from pixel's neighborhood.

When you apply this many times you lose specific pixel's information, but it gains information about pixel's neighborhood. And this is good because maybe this neighborhood is a corner, a line, curve, etc. (this is what learned the kernel to identify value content, for that reason the network in d and e have more basic forms and color without detail) . if you continue applying more convolutions you will find a abstract representation of the all the input, even you can get only one number that represent that input (if you don't avoid dimension reduction with padding or other tool) and say something like "oh it's a five and therefore the input is a house" (in classification case).

NOTE: This is very general and many details were omitted, if you want to learn more I recommend the book deep learning (Goodfellow et al., 2016) chapter 9 convolutional networks.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.