I'm trying to understand the intuition behind how the Content Loss is calculated in a Neural Style Transfer. I'm reading from an articles: https://medium.com/mlreview/making-ai-art-with-style-transfer-using-keras-8bb5fa44b216 , that explains the implementation of Neural Style Transfer, from the Content loss function:
The article explains that:
F and P are matrices with a number of rows equal to N and a number of columns equal to M.
N is the number of filters in layer l and M is the number of spatial elements in the feature map (height times width) for layer l.
From the code below for getting the features/content representation from particular Conv layers, I didn't quite understand how it works. Basically I printed out the output of every line of code to try to make it easier, but it still left a number of questions to be asked, which I listed below the code:
def get_feature_reps(x, layer_names, model): """ Get feature representations of input x for one or more layers in a given model. """ featMatrices =  for ln in layer_names: selectedLayer = model.get_layer(ln) featRaw = selectedLayer.output featRawShape = K.shape(featRaw).eval(session=tf_session) N_l = featRawShape[-1] M_l = featRawShape*featRawShape featMatrix = K.reshape(featRaw, (M_l, N_l)) featMatrix = K.transpose(featMatrix) featMatrices.append(featMatrix) return featMatrices def get_content_loss(F, P): cLoss = 0.5*K.sum(K.square(F - P)) return cLoss
1- For the line
featRaw = selectedLayer.output, when I print
featRaw, I get the output:
Tensor("block4_conv2/Relu:0", shape=(1, 64, 64, 512), dtype=float32).
Relu:0does this mean Relu activation has not yet been applied?
b- Also I presume we're outputing the feature maps outputs from
block4_conv2, not the filters/kernels themselves, correct?
c- Why is there an axis of 1 at the start? My understanding of Conv layers is that they're simply made up from the number of filters/kernels (with shape-height, width, depth) to apply to the input.
selectedLayer.outputsimply outputs the shape of the Conv layer, or does the output object also hold other information like the pixel values from the output feature maps of the layer?
2- With the line:
featMatrix = K.reshape(featRaw, (M_l, N_l) where printing
featMatrix would output:
Tensor("Reshape:0", shape=(4096, 512), dtype=float32).
a- This is where I'm confused the most. So to get the feature/content representation of a particular Conv layer of an image, we simply create a matrix of 2 dimensions, the first being the number of filters and the other being the area of the filter/kernel (height * width). That doesn't make sense! How do we get unique feature of an image from just that?!! We're not retrieving any pixel values from a feature map. We're simply getting the area size of filter/kernel and the number of filters, but not retrieving any of the content (pixel values) itself!!
b- Also the final
featMatrixis transposed - i.e.
featMatrix = K.transpose(featMatrix)with the output
Tensor("transpose:0", shape=(512, 4096), dtype=float32). Why is that (i.e. why reverse the axis)?
3 - Finally I want to know, once we retrieve the content representation, how can I output that in both as a numpy array and save it as an image?
Any help would be really appreciated.