Can someone explain R1 regularization function in simple terms?

Question

I'm trying to understand the R1 regularization function, both the abstract concept and every symbol in the formula. According to the article, the definition of R1 is:

It penalizes the discriminator from deviating from the Nash Equilibrium via penalizing the gradient on real data alone: when the generator distribution produces the true data distribution and the discriminator is equal to 0 on the data manifold, the gradient penalty ensures that the discriminator cannot create a non-zero gradient orthogonal to the data manifold without suffering a loss in the GAN game.

$R_1(\psi ) = \frac{\gamma}{2}E_{pD(x)}\left [ \left \| \bigtriangledown D_{\psi}(x) \right \|^2 \right ]$

I have basic understanding of how GAN's and back-propagation works. I understand the idea of punishing the discriminator when he deviates from the Nash equilibrium. The rest of it gets murky, even if it might be basic math. For example, I'm not sure why it matters if the gradient is orthogonal to the data.

On the equation part, it's even more unclear. The discriminator input is always an image, so I assume $x$ is an image. Then what is $\psi$ and $\gamma$?

(I understand this is somewhat of a basic question, but seems there are no blogs about it for us simple non-researchers, math challenged people who fail to understand the original article )

Answer 1

Here is how I understand this regularization.

$R_1$ is simply the norm of the gradients, which indicates how fast the weights will be updated. Gradient regularization penalizes large changes in the output of some neural network layer.

$$ R_{1}\left(\psi\right) = \frac{\gamma}{2}E_{p_{D}\left(x\right)}\left[||\nabla{D_{\psi}\left(x\right)}||^{2}\right]\text{,} $$

where $\psi$ is discriminator weights, $E_{p_{D}\left(x\right)}$ means that we sample data only form the real distribution (i.e. only real images) and $\gamma$ is a hyperparameter.

Since we don't know if $G$ can already generate data from real distribution, we apply this regularization to $D$ only on real data, because we don't want the discriminator to create a non-zero gradient without suffering a loss if we are already in a Nash Equilibrium. I guess this also prevents $G$ from updating if it generates data from the real distribution.

The authors also investigate which value is best for $\gamma$ by analyzing the eigenvalues of the Jacobian of the the associated gradient vector field, but in my opinion, this value is highly dependent on dataset and architecture.

"Gradient orthogonal to the data manifold" simply means zero gradients. From a GAN perspective, the data manifold is a lower-dimensional latent features manifold embedded in a higher-dimensional space and our goal is to approximate it. Since the gradient vector shows the direction in which we need to update our function, if it is orthogonal to this manifold, we do not need to update the function.