I am currently in the process of reading and understanding the process of style transfer. I came across this equation in the research paper which went like - enter image description here

For context, here is the paragraph -

Generally each layer in the network defines a non-linear filter bank whose complexity increases with the position of the layer in the network. Hence a given input image is encoded in each layer of the Convolutional Neural Network by the filter responses to that image. A layer with $N_l$ distinct filters has $N$ feature maps each of size $M$ , where $M_l$ is the height times the width of the feature map. So the re- sponses in a layer l can be stored in a matrix $Fl ∈ R^{N_l×M_l}$ where F l is the activation of the ith filter at position j in ij layer l. To visualise the image information that is encoded at different layers of the hierarchy one can perform gradient descent on a white noise image to find another image that matches the feature responses of the original image (Fig 1, content reconstructions). Let $\vec p$ and $\vec x$ be the original image and the image that is generated, and $P^l$ and $F^l$ their respective feature representation in layer l. We then define the squared-error loss between the two feature representations $\mathcal{L_{content}(\vec p, \vec x, l)} = {1\over 2} \Sigma_{i,j} \big(F_{ij}^l - P_{ij}^l \big)$. The derivative of this loss with respect to the activations in layer $l$ [the equation above $(2)$].

I just want to know why the partial derivative is $0$ when $F_{ij}^l < 0$.

  • $\begingroup$ Hello. Please, put your specific question in the title. "Math behind style transfer" is very vague/general and not a question. Thank you! $\endgroup$
    – nbro
    Aug 13 '21 at 13:10
  • $\begingroup$ K I actually tried to sum up my entire Q in that one line - wasnt able to in the beginning XD. I'll do it soon $\endgroup$
    – HarshDarji
    Aug 13 '21 at 13:18

$F_l$ is the activation of the filter. They state in the paper that they base their method on VGG-Network, which uses ReLU as its activation function. In fact, VGG uses it in all of its hidden layers. ReLU is defined as

$$f(x) = max(0,x)$$

Since ReLU is 0 for all x's below 0, the equation above holds; When x is non-positive, all terms in the loss function are constants with respect to $F_{ij}^l$.

  • $\begingroup$ Ohh okay. Thanks! $\endgroup$
    – HarshDarji
    Aug 13 '21 at 12:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.