5
$\begingroup$

I have been reading the paper which introduced spectral normalization in GANs.

At some point the paper mentions the following:

The machine learning community has been pointing out recently that the function space from which the discriminators are selected crucially affects the performance of GANs. A number of works (Uehara et al., 2016; Qi, 2017; Gulrajani et al., 2017) advocate the importance of Lipschitz continuity in assuring the boundedness of statistics.

What does it mean that the Lipschitz continuity assures the boundedness of statistics and why does that happen?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

To put it simply GANs suffer from a problem of uneven learning rate. Imagine the learning rate of a pitcher and hitter if the pitcher gets to a point where they can throw much better than the hitter can hit then the hitter may fall into a 'training pit' as to be unable to ever learn how to hit from the pitcher.

This follows a continues relationship in between the two learning rates where if the pitcher is becoming a much better pitcher at a faster rate they could become too good and make learning impossible for the hitter. So the rate must be 'slowed down' as to ensure the pitcher doesn't ruin the hitter.

If the cone of the Lipschitz continuity function of either function is outrunning/outpacing the other than the learning for the one who is in front must be slowed down so the other catches up.

Two runners trying to push each other athletically is another example. If one outpaces the other an injury may occur in the one lagging behind while trying to keep pace this happens, when the adversarial network becomes too good at generating training material that the behind network is not ready to learn with.

The GAN will do best when the learning rates are adjusted to slow down the fast learner artificially.

The statistics will not be bounded correctly if the learning rates are not kept in check similarly to how step size needs to be right to find local minimum and maximum. If the learning is not artificial augmented so both keep relatively same pace getting stuck at local minimum and maximum of the solution space will occur.

Improve this answer
$\endgroup$
2
  • $\begingroup$ What do you mean by saying that the cone of one function is outrunning the other? I know that for a Lipschitz function there should exist a cone which you can move along the x axis and have the whole graph outside the cone, but I do not understand what do you mean by this. $\endgroup$
    – MattSt
    Commented Nov 15, 2019 at 14:20
  • $\begingroup$ If the two functions representing the two problems learning space differ and the slope of one is steeper then the other then the learning rates will be unequal. If one of the models out runs the other than it wrecks the two runners keeping pace phenomenon. For learning problems a function exists that represents different phases of learning the problem. Like a discriminators may have a nearly flat line learning the difference between black boots and white boots but may have a very steep(difficult) time learning the difference between black stilettos and black pumps. $\endgroup$
    – Michael Hearn
    Commented Nov 15, 2019 at 16:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .