10
$\begingroup$

I'm trying to understand the R1 regularization function, both the abstract concept and every symbol in the formula. According to the article, the definition of R1 is:

It penalizes the discriminator from deviating from the Nash Equilibrium via penalizing the gradient on real data alone: when the generator distribution produces the true data distribution and the discriminator is equal to 0 on the data manifold, the gradient penalty ensures that the discriminator cannot create a non-zero gradient orthogonal to the data manifold without suffering a loss in the GAN game.

$R_1(\psi ) = \frac{\gamma}{2}E_{pD(x)}\left [ \left \| \bigtriangledown D_{\psi}(x) \right \|^2 \right ]$

I have basic understanding of how GAN's and back-propagation works. I understand the idea of punishing the discriminator when he deviates from the Nash equilibrium. The rest of it gets murky, even if it might be basic math. For example, I'm not sure why it matters if the gradient is orthogonal to the data.

On the equation part, it's even more unclear. The discriminator input is always an image, so I assume $x$ is an image. Then what is $\psi$ and $\gamma$?

(I understand this is somewhat of a basic question, but seems there are no blogs about it for us simple non-researchers, math challenged people who fail to understand the original article )

$\endgroup$
3
  • 1
    $\begingroup$ Rather than providing a link to an external resource, can you just type in latex (yes, you can use latex on this site) of the specific formula that you're referring to? Moreover, please, provide some context, and do not just say "explain X". $\endgroup$
    – nbro
    Dec 30, 2020 at 10:43
  • 1
    $\begingroup$ @nbro I added latex. I initially avoided adding context because I wanted to keep this question more generic as this really seems to be mostly uncovered by popular blogs. I tried to add some directions but I'm afraid I don't have any more concrete context other than I encountered that StyleGAN uses for one dataset and not for another, and I want to "get a feel" as to how it works. $\endgroup$ Dec 30, 2020 at 11:31
  • $\begingroup$ Thank you! Even if you're looking for a general answer, providing some context is always a good idea. $\endgroup$
    – nbro
    Dec 30, 2020 at 11:45

2 Answers 2

8
$\begingroup$

Here is how I understand this regularization (updated).

$R_1$ is simply the norm of the gradients w.r.t. to the real data (gradients indicate the direction and magnitude by which the weight should be updated during training. So, $R_1$ prevents large weights update for the discriminator if it has high gradients on real data, effectively smoothing its decision boundary around the real data points.

$$ R_{1}\left(\psi\right) = \frac{\gamma}{2}E_{p_{D}\left(x\right)}\left[||\nabla{D_{\psi}\left(x\right)}||^{2}\right]\text{,} $$

where $\psi$ is discriminator weights, $E_{p_{D}\left(x\right)}$ means that we sample data only from the real distribution (i.e. only real images) and $\gamma$ is a hyperparameter.

The intuition is that we want the discriminator's output to be minimally sensitive to small changes in the real data distribution. This helps prevent overfitting to particular noise in the training data. It encourages the discriminator to be less confident overall, thus indirectly helping the generator to improve by preventing the discriminator from becoming too perfect too quickly (a very confident discriminator is a known problem for KL-loss).

The authors also investigate which value is best for $\gamma$ by analyzing the eigenvalues of the Jacobian of the associated gradient vector field, but in my opinion, this value is highly dependent on the dataset and architecture.

When we say "gradient orthogonal to the data manifold," we are referring to changes in the discriminator's parameters that would affect its decision-making process in ways that are not along the data manifold. Since the data manifold represents the set of points in the high-dimensional space that corresponds to real data, gradients in directions orthogonal to this manifold don't improve the discriminator's ability to classify the real data correctly; such gradients might signify overfitting or instability in training.

$\endgroup$
1
$\begingroup$

The current accepted answer is completely incorrect. The $R_1$ regularization is dealing with the gradient with respect to the data $x$, not the model parameters. The gradient being orthogonal to the data manifold doesn't even make sense if we are talking about the gradient with respect to the parameters.

With that out of the way, the essential idea behind $R_1$ normalization is that you want to "flatten" the response of the discriminator. That is, you don't want the value of the discriminator to change too quickly between similar data points. This is a problem because when your generator starts to get good and produce realistic points, your discriminator starts panicking and getting sensitive and paranoid about realistic data points, since the generator is such a good imposter. So the discriminator might classify one datapoint as real, but an almost identical datapoint as fake. You don't want this as it leads to unhelpful extreme feedback from the discriminator so we employ the $R_1$ regulizer, which is a form of gradient penalty to smooth out the discriminator output.

Now let's talk about the gradient being orthogonal to the data. The key point here is the context. The generator is producing "good" samples, and the discriminator is 0 on the real data, meaning the generator is doing so well that the discriminator has essentially no skill yet and claims everything is fake. A non-zero gradient orthogonal to the data means that the discriminator thinks that as you go in the direction towards unrealistic data (orthogonal), then I will mistakenly think that data is more realistic. The gradient penalty helps stop this by squashing the gradients on the data manifold.

$\endgroup$
1
  • $\begingroup$ Good points, thanks. I've updated my answer $\endgroup$ Jan 11 at 15:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .