Understanding the intuition behind Content Loss (Neural Style Transfer) [closed]

Question

I'm trying to understand the intuition behind how the Content Loss is calculated in a Neural Style Transfer. I'm reading from an articles: https://medium.com/mlreview/making-ai-art-with-style-transfer-using-keras-8bb5fa44b216 , that explains the implementation of Neural Style Transfer, from the Content loss function:

The article explains that:

• F and P are matrices with a number of rows equal to N and a number of columns equal to M.

• N is the number of filters in layer l and M is the number of spatial elements in the feature map (height times width) for layer l.

From the code below for getting the features/content representation from particular Conv layers, I didn't quite understand how it works. Basically I printed out the output of every line of code to try to make it easier, but it still left a number of questions to be asked, which I listed below the code:

def get_feature_reps(x, layer_names, model):
    """
    Get feature representations of input x for one or more layers in a given model.
    """
    featMatrices = []
    for ln in layer_names:
        selectedLayer = model.get_layer(ln)
        featRaw = selectedLayer.output
        featRawShape = K.shape(featRaw).eval(session=tf_session)
        N_l = featRawShape[-1]
        M_l = featRawShape[1]*featRawShape[2]
        featMatrix = K.reshape(featRaw, (M_l, N_l))
        featMatrix = K.transpose(featMatrix)
        featMatrices.append(featMatrix)
    return featMatrices

def get_content_loss(F, P):
    cLoss = 0.5*K.sum(K.square(F - P))
    return cLoss

1- For the line featRaw = selectedLayer.output, when I print featRaw, I get the output: Tensor("block4_conv2/Relu:0", shape=(1, 64, 64, 512), dtype=float32).

• a- Relu:0 does this mean Relu activation has not yet been applied?

• b- Also I presume we're outputing the feature maps outputs from block4_conv2, not the filters/kernels themselves, correct?

• c- Why is there an axis of 1 at the start? My understanding of Conv layers is that they're simply made up from the number of filters/kernels (with shape-height, width, depth) to apply to the input.

• d- Is selectedLayer.output simply outputs the shape of the Conv layer, or does the output object also hold other information like the pixel values from the output feature maps of the layer?

2- With the line: featMatrix = K.reshape(featRaw, (M_l, N_l) where printing featMatrix would output: Tensor("Reshape:0", shape=(4096, 512), dtype=float32).

• a- This is where I'm confused the most. So to get the feature/content representation of a particular Conv layer of an image, we simply create a matrix of 2 dimensions, the first being the number of filters and the other being the area of the filter/kernel (height * width). That doesn't make sense! How do we get unique feature of an image from just that?!! We're not retrieving any pixel values from a feature map. We're simply getting the area size of filter/kernel and the number of filters, but not retrieving any of the content (pixel values) itself!!

• b- Also the final featMatrix is transposed - i.e. featMatrix = K.transpose(featMatrix) with the output Tensor("transpose:0", shape=(512, 4096), dtype=float32). Why is that (i.e. why reverse the axis)?

3 - Finally I want to know, once we retrieve the content representation, how can I output that in both as a numpy array and save it as an image?

Any help would be really appreciated.

Answer 1

1. For the line featRaw = selectedLayer.output, when I print featRaw, I get the output: Tensor("block4_conv2/Relu:0", shape=(1, 64, 64, 512), dtype=float32).

   • a) Relu:0 does this mean Relu activation has not yet been applied?

It has been applied.

• b) Also I presume we're outputing the feature maps outputs from block4_conv2, not the filters/kernels themselves, correct?

Yes.

• c) Why is there an axis of 1 at the start? My understanding of Conv layers is that they're simply made up from the number of filters/kernels (with shape-height, width, depth) to apply to the input.

The first dimension in keras is used for representing batch dimension. When training the model, you aren't passing the images one by one but in batches. The number of images that are processed in parallel before the model calculates its loss and makes its update is called the batch_size. When computing the content loss, though, you have a single image you want to perform style transfer to, so batch_size=1. That's why you see the unit in the beginning of the tensor's shape.

• d) Is selectedLayer.output simply outputs the shape of the Conv layer, or does the output object also hold other information like the pixel values from the output feature maps of the layer?

It outputs the tensor containing all the pixel values that are outputted from that layer.

2. With the line: featMatrix = K.reshape(featRaw, (M_l, N_l) where printing featMatrix would output: Tensor("Reshape:0", shape=(4096, 512), dtype=float32).

   • a) This is where I'm confused the most. So to get the feature/content representation of a particular Conv layer of an image, we simply create a matrix of 2 dimensions, the first being the number of filters and the other being the area of the filter/kernel (height * width). That doesn't make sense! How do we get unique feature of an image from just that?!! We're not retrieving any pixel values from a feature map. We're simply getting the area size of filter/kernel and the number of filters, but not retrieving any of the content (pixel values) itself!!

I think you're confusing things a bit. You are getting the pixel values, just in a different shape. Instead of stacking them in a $64 \times 64 \times 515$ array you are stacking them in a $1096 \times 515$ array. Exactly the same information is stored in a different shape. Think of it like transforming a $10 \times 2$ matrix to a $5 \times 4$ one. It contains the same information, in a different shape.

• b) Also the final featMatrix is transposed - i.e. featMatrix = K.transpose(featMatrix) with the output Tensor("transpose:0", shape=(512, 4096), dtype=float32). Why is that (i.e. why reverse the axis)?

I'm not sure why exactly he does this, but from a bit that I saw the code I think it has something to do with the style loss (not the content loss). Probably something didn't match up with the dimensions of some operation, maybe a matrix multiplication...

3. Finally I want to know, once we retrieve the content representation, how can I output that in both as a numpy array and save it as an image?

If you have the output image as a tensor, let's say y. You can convert it to numpy through keras.backend.eval:

import keras.backend as K

y_arr = K.eval(y)

There are many ways to save the image, including scikit-image, scipy, opencv, PIL, matplotlib, etc. I'll use the last:

import matplotlib.pyplot as plt

plt.imsave('my_image.png', y_arr)

Note that in order to work y_arr should have a shape of either (height, width, 3) (if it is an RGB image) or (height, width) (if it is grayscale).