I've recently been reading a lot about style transfer, its applications and implications. I understand what the Gram matrix is and does. I can program it. But one thing that has been boggling me is: how does the VGG style loss incorporate color information into the style?

In the paper "Texture Synthesis by CNNs", Gatys et al. show that minimizing the MSE between the Gram matrices of a random white noise image and a "target texture" yields new instances of that texture, with stochastic variation. I understand that this must work, as the Gram matrix measures the correlation between features detected by the VGG activations across channels, without spatial relation. So if we optimize the white noise image to have the same Gram matrix, it will exhibit the same statistics, and hence look like an instance of the original texture.

But how does this work with color? Of course, the VGG could learn something like a mean filter, with all ones, whose output would be the avg. color over that filter kernel. After all, "color" is just another statistic. But then when using that in conjunction with the Gram loss, wouldn't this information be lost, as it's all just correlation and hence "relative" to each other?

While writing this question, I'm starting to think of it like this: Maybe the feature correlation expresses these color constraints in some form like: "if one part is red, there must be a green part close to it" (for the radish), or "if there is a rounded edge, one side of it must be in shadow (=darker)" in case of the stone texture. This would tie color to the surrounding statistics (e.g., edges, other colors) and is the only reason I can think of why this works at all.

Can somebody confirm/refute this, and share their thoughts? Happy to discuss!

Image Source: Gatys et al., Texture Synthesis by Convolutional Neural Networks

Image Source: Texture Synthesis by Convolutional Neural Networks, Gatys et al.


My two cents on this topic:

"After all, "color" is just another statistic", I think this is the simple (and correct) answer to the question. To go a bit deeper, you can check this paper, which shows how minimizing a loss based on the Gram matrix is mathematically equivalent to minimizing the Maximum Mean Discrepancy between the inputs and targets distribution. The two distributions inevitably contains information about colors, so while disentangle spatial features is rather simple (you could simply show one pixel at the time instead of an image), disentangling colors is much more tricky, cause it's an intrinsic characteristic of each point.

A last remark from my side is that the main problem when working with style transfer is precisely that "style" mean everything. This is not a problem for papers that simply try to achieve it per se, i.e. without a real use case in mind, but it becomes fundamental in real applications. A concrete example of this is super resolution. Many papers try to achieve it with style transfer, coupling low resolution and high resolution images. Ideally the features you would like to transfer are enhanced sharpness and maybe texture injection for details generation. Problem is that along with them there are always side features that hinder the quality of the resulting image, among which noise specific to the target domain, and also colors.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.