2
$\begingroup$

I'm trying to understand the intuition behind how the Content Loss is calculated in a Neural Style Transfer. I'm reading from an articles: https://medium.com/mlreview/making-ai-art-with-style-transfer-using-keras-8bb5fa44b216 , that explains the implementation of Neural Style Transfer, from the Content loss function: enter image description here

The article explains that:

  • F and P are matrices with a number of rows equal to N and a number of columns equal to M.

  • N is the number of filters in layer l and M is the number of spatial elements in the feature map (height times width) for layer l.

From the code below for getting the features/content representation from particular Conv layers, I didn't quite understand how it works. Basically I printed out the output of every line of code to try to make it easier, but it still left a number of questions to be asked, which I listed below the code:

def get_feature_reps(x, layer_names, model):
    """
    Get feature representations of input x for one or more layers in a given model.
    """
    featMatrices = []
    for ln in layer_names:
        selectedLayer = model.get_layer(ln)
        featRaw = selectedLayer.output
        featRawShape = K.shape(featRaw).eval(session=tf_session)
        N_l = featRawShape[-1]
        M_l = featRawShape[1]*featRawShape[2]
        featMatrix = K.reshape(featRaw, (M_l, N_l))
        featMatrix = K.transpose(featMatrix)
        featMatrices.append(featMatrix)
    return featMatrices

def get_content_loss(F, P):
    cLoss = 0.5*K.sum(K.square(F - P))
    return cLoss

1- For the line featRaw = selectedLayer.output, when I print featRaw, I get the output: Tensor("block4_conv2/Relu:0", shape=(1, 64, 64, 512), dtype=float32).

  • a- Relu:0 does this mean Relu activation has not yet been applied?

  • b- Also I presume we're outputing the feature maps outputs from block4_conv2, not the filters/kernels themselves, correct?

  • c- Why is there an axis of 1 at the start? My understanding of Conv layers is that they're simply made up from the number of filters/kernels (with shape-height, width, depth) to apply to the input.

  • d- Is selectedLayer.output simply outputs the shape of the Conv layer, or does the output object also hold other information like the pixel values from the output feature maps of the layer?

2- With the line: featMatrix = K.reshape(featRaw, (M_l, N_l) where printing featMatrix would output: Tensor("Reshape:0", shape=(4096, 512), dtype=float32).

  • a- This is where I'm confused the most. So to get the feature/content representation of a particular Conv layer of an image, we simply create a matrix of 2 dimensions, the first being the number of filters and the other being the area of the filter/kernel (height * width). That doesn't make sense! How do we get unique feature of an image from just that?!! We're not retrieving any pixel values from a feature map. We're simply getting the area size of filter/kernel and the number of filters, but not retrieving any of the content (pixel values) itself!!

  • b- Also the final featMatrix is transposed - i.e. featMatrix = K.transpose(featMatrix) with the output Tensor("transpose:0", shape=(512, 4096), dtype=float32). Why is that (i.e. why reverse the axis)?

3 - Finally I want to know, once we retrieve the content representation, how can I output that in both as a numpy array and save it as an image?

Any help would be really appreciated.

$\endgroup$
1
$\begingroup$
  1. For the line featRaw = selectedLayer.output, when I print featRaw, I get the output: Tensor("block4_conv2/Relu:0", shape=(1, 64, 64, 512), dtype=float32).

    • a) Relu:0 does this mean Relu activation has not yet been applied?

It has been applied.

  • b) Also I presume we're outputing the feature maps outputs from block4_conv2, not the filters/kernels themselves, correct?

Yes.

  • c) Why is there an axis of 1 at the start? My understanding of Conv layers is that they're simply made up from the number of filters/kernels (with shape-height, width, depth) to apply to the input.

The first dimension in keras is used for representing batch dimension. When training the model, you aren't passing the images one by one but in batches. The number of images that are processed in parallel before the model calculates its loss and makes its update is called the batch_size. When computing the content loss, though, you have a single image you want to perform style transfer to, so batch_size=1. That's why you see the unit in the beginning of the tensor's shape.

  • d) Is selectedLayer.output simply outputs the shape of the Conv layer, or does the output object also hold other information like the pixel values from the output feature maps of the layer?

It outputs the tensor containing all the pixel values that are outputted from that layer.

  1. With the line: featMatrix = K.reshape(featRaw, (M_l, N_l) where printing featMatrix would output: Tensor("Reshape:0", shape=(4096, 512), dtype=float32).

    • a) This is where I'm confused the most. So to get the feature/content representation of a particular Conv layer of an image, we simply create a matrix of 2 dimensions, the first being the number of filters and the other being the area of the filter/kernel (height * width). That doesn't make sense! How do we get unique feature of an image from just that?!! We're not retrieving any pixel values from a feature map. We're simply getting the area size of filter/kernel and the number of filters, but not retrieving any of the content (pixel values) itself!!

I think you're confusing things a bit. You are getting the pixel values, just in a different shape. Instead of stacking them in a $64 \times 64 \times 515$ array you are stacking them in a $1096 \times 515$ array. Exactly the same information is stored in a different shape. Think of it like transforming a $10 \times 2$ matrix to a $5 \times 4$ one. It contains the same information, in a different shape.

  • b) Also the final featMatrix is transposed - i.e. featMatrix = K.transpose(featMatrix) with the output Tensor("transpose:0", shape=(512, 4096), dtype=float32). Why is that (i.e. why reverse the axis)?

I'm not sure why exactly he does this, but from a bit that I saw the code I think it has something to do with the style loss (not the content loss). Probably something didn't match up with the dimensions of some operation, maybe a matrix multiplication...

  1. Finally I want to know, once we retrieve the content representation, how can I output that in both as a numpy array and save it as an image?

If you have the output image as a tensor, let's say y. You can convert it to numpy through keras.backend.eval:

import keras.backend as K

y_arr = K.eval(y)

There are many ways to save the image, including scikit-image, scipy, opencv, PIL, matplotlib, etc. I'll use the last:

import matplotlib.pyplot as plt

plt.imsave('my_image.png', y_arr)

Note that in order to work y_arr should have a shape of either (height, width, 3) (if it is an RGB image) or (height, width) (if it is grayscale).

$\endgroup$
  • $\begingroup$ Thank you so much for the response and clarification. Much appreciated. I did the K.eval(P) and it worked. P is the output content representation tensor: P = get_feature_reps(x=cImArr, layer_names=[cLayerName], model=cModel)[0]. Using K.eval(P) I got an output dtype of float32 and a shape of (512, 4096). So just referring to your very last answer, for ensuring to convert to the correct format for RGB image, how do I convert a shape of (512, 4096) to the correct shape of (height, width, 3). $\endgroup$ – Hazzaldo Aug 30 at 23:10
  • $\begingroup$ When I saved the array to an image with the shape (512, 4096), the image looked completely weird: imgur.com/XHUlqtY It looks nothing like the original image: imgur.com/oZrBlX0 $\endgroup$ – Hazzaldo Aug 30 at 23:10
  • $\begingroup$ My second follow up question is, the eval() function did not work on the style representation tensor data structure (As), for this code output: As = get_feature_reps(x=sImArr, layer_names=sLayerNames, model=sModel). $\endgroup$ – Hazzaldo Aug 30 at 23:11
  • 1
    $\begingroup$ You first need to transpose the array to (4096, 512). You can do this by y_arr = y_arr.T. Then you need to reshape it to the original dimensions: y_arr = y_arr.reshape((64, 64, 512)). I don't think you can turn these into rgb images. I think this represents 512 grayscale feature maps. To get each you can simply slice the array, e.g. y_arr[:, :, 13] (for the image with index 13). $\endgroup$ – Djib2011 Aug 31 at 23:38
  • 1
    $\begingroup$ You can think of K.eval() as a method of converting tensors to arrays. It can't be applied to As because As is a list not a tensor. Instead it can be applied to the contents of the list, which happen to be tensors, e.g. K.eval(As[0]) for the first tensor from the list. $\endgroup$ – Djib2011 Aug 31 at 23:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.