I am currently in the process of reading and understanding the process of style transfer. I came across this equation in the research paper which went like - enter image description here

For context, here is the paragraph -

Generally each layer in the network defines a non-linear filter bank whose complexity increases with the position of the layer in the network. Hence a given input image is encoded in each layer of the Convolutional Neural Network by the filter responses to that image. A layer with $N_l$ distinct filters has $N$ feature maps each of size $M$ , where $M_l$ is the height times the width of the feature map. So the re- sponses in a layer l can be stored in a matrix $Fl ∈ R^{N_l×M_l}$ where F l is the activation of the ith filter at position j in ij layer l. To visualise the image information that is encoded at different layers of the hierarchy one can perform gradient descent on a white noise image to find another image that matches the feature responses of the original image (Fig 1, content reconstructions). Let $\vec p$ and $\vec x$ be the original image and the image that is generated, and $P^l$ and $F^l$ their respective feature representation in layer l. We then define the squared-error loss between the two feature representations $\mathcal{L_{content}(\vec p, \vec x, l)} = {1\over 2} \Sigma_{i,j} \big(F_{ij}^l - P_{ij}^l \big)$. The derivative of this loss with respect to the activations in layer $l$ [the equation above $(2)$].

I just want to know why the partial derivative is $0$ when $F_{ij}^l < 0$.

  • $\begingroup$ Hello. Please, put your specific question in the title. "Math behind style transfer" is very vague/general and not a question. Thank you! $\endgroup$
    – nbro
    Aug 13, 2021 at 13:10
  • $\begingroup$ K I actually tried to sum up my entire Q in that one line - wasnt able to in the beginning XD. I'll do it soon $\endgroup$
    – HarshDarji
    Aug 13, 2021 at 13:18

1 Answer 1


$F_l$ is the activation of the filter. They state in the paper that they base their method on VGG-Network, which uses ReLU as its activation function. In fact, VGG uses it in all of its hidden layers. ReLU is defined as

$$f(x) = max(0,x)$$

Since ReLU is 0 for all x's below 0, the equation above holds; When x is non-positive, all terms in the loss function are constants with respect to $F_{ij}^l$.

  • $\begingroup$ Ohh okay. Thanks! $\endgroup$
    – HarshDarji
    Aug 13, 2021 at 12:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .