In style transfer, why does the comparison between channels give a good sense of style?

I have been learning about Style Transfer recently. Style is defined as

The correlation of activations between channels.

I can't seem to understand why that would be true. Intuitively, style seems to be the patterns that exist in one particular channel/image rather than the patterns between channels. When filters in CNNs have different weights for acting as filters for different channels, why do we even expect 2 channels to be correlated? And further, why do we expect the style to be conveyed by it?

I expected a style function that could compare activations in some layer of a CNN condensed into one channel so that an algorithm can search for which activations occur simultaneously and hold style information.

I understand how we are carrying out the operations with the matrix and defining the loss function, what I don't get is why we are assuming style information lies in correlation between channels in the first place.

When the original Neural Transfer paper was published, your question stayed unanswered for a while. The reason why the Gram matrix represents artistic style was not entirely clear. A satisfactory (in my opinion) explanation came with the "Demystifying Neural Style Transfer" paper.

The basic idea is that you cannot just directly compare activations for two images. The spatial positions of various features are different for the source and target image, so you should somehow get rid of positional information of the activations and compare their distributions across the whole image. The goal of the style transfer task is thus to make the two distributions as close as possible. One of the possible measures of distance between two distributions $$P$$ and $$Q$$ is the Maximum Mean Distance (MMD):

$$\text{MMD}^2(P,Q) = \left\Vert \mathbb{E}_{P}[\phi(X)] - \mathbb{E}_{Q}[\phi(Y)]\right\Vert^2$$

With $$\phi(\cdot)$$ being a feature function - in our case it would be the NN activations in a particular layer. The next step would be to apply a "kernel trick" to the MMD, representing it through a kernel $$k(x,y)$$

$$\text{MMD}^2(P,Q) = \mathbb{E}[k(x_i,x_j)] + \mathbb{E}[k(y_i,y_j)] - 2 \mathbb{E}[k(x_i,y_j)]$$

The Gram matrix of the original style transfer corresponds to the squared dot-product kernel $$k(x,y) = (x^Ty)^2$$: $$\text{MMD}^2(P,Q) = \mathbb{E}[(x^T_ix_j)^2] + \mathbb{E}[(y^T_iy_j)^2] - 2 \mathbb{E}[(x^T_iy_j)^2] = \left\Vert G^x - G^y\right\Vert_F^2$$

Where $$G^x$$ and $$G^y$$ are the Gram matrices for activations and $$\Vert\cdot\Vert_F$$ is the Frobenius norm. The last equality gives a bit of trouble to those not used to the kernel trick, so I'll expand it (assuming Einstein summation convention):

$$\begin{array}{l} \mathbb{E}[(x^T_ix_j)^2] + \mathbb{E}[(y^T_iy_j)^2] - 2 \mathbb{E}[(x^T_iy_j)^2] = \\ = x_{ik}x_{jk}x_{im}x_{jm} + y_{ik}y_{jk}y_{im}y_{jm} - 2 x_{ik}x_{jk}y_{im}y_{jm} \\ = (x_{ik}x_{im} - y_{ik}y_{im})(x_{jk}x_{jm} - y_{jk}y_{jm}) \\ = (G^x_{km} - G^y_{km})(G^x_{km} - G^y_{km}) \\ = \left\Vert G^x - G^y\right\Vert_F^2 \end{array}$$

The authors of the 1701.01036 also tried different kernel functions $$k(x,y)$$ getting more interesting style transfer results.

• Hi, thanks for the thorough reply, just need a couple of clarifications to allow me to fully understand this. The MMD^2 portion you've written is basically a squared L2 Norm applied to the feature vectors of X and Y that are unrolled and then subtracted, correct? Also, I am sorry if I am unfamiliar with common notation but what does the E represent? May 17 '21 at 22:57
• @JaskeeratSinghSarin The $\mathbb{E}$ represents "in expectation"- it assumes some population or probability distribution of observed variables. The basic equation you can bear in mind when converting to/from expectations is that if you have random variable $X$ that follows discrete probability distribution $p(x) = \mathbf{Pr}\{X=x\}$, then $\mathbb{E}[X] = \sum_{x} p(x)x$ May 20 '21 at 7:21