1
$\begingroup$

Most basic GAN algorithm definitions I found go like this:

  1. Generate
  2. Train discriminator
  3. Generate
  4. Train generator

Like this one: GAN pseudocode

If I'm not misunderstanding, we sample twice from the same generator in a single iteration due to the fakes being generated before the weights are updated for the generator. Can you tell me why we need to sample twice?

$\endgroup$

1 Answer 1

1
$\begingroup$

By resampling from the generator after each update to the discriminator, you ensure that the generator is adapting to the evolving discriminator and producing diverse and realistic fake samples in response to the current state of the discriminator. If you sampled only once in one iteration, the generator would be updated based on the same set of fake samples used in the previous discriminator update, leading to potential mode collapse where the generator's training in the same iteration after discriminator's training becomes overly focused on a subset of possible outputs and fails to produce diverse and realistic samples and losing its adversarial training nature, essentially just a slowly improved actor always reacting to an unfriendly critic.

$\endgroup$
2
  • 1
    $\begingroup$ Yeah, I arrived at a similar conclusion. The discriminator would have an 'unfair' advantage as it would have experienced the samples already when evaluating the generator loss. I do still wonder how much of an issue it is practically, as I do not have a good intuition about how much of an advantage a single exposure is, especially at low learning rates. But I guess that's a question that no one really tested given that it conceptually doesn't make sense to omit the second sampling anyways. $\endgroup$
    – John Smith
    Commented Mar 12 at 7:45
  • $\begingroup$ GAN's generator is always deterministic to directly output synthetic data capturing only their rough distribution without (conditional) probabilities like VAEs, so intuitively mode collapse is a much more serious problem for GAN's generator and a second random noise as exploration in the same adversarial iteration definitely helps. $\endgroup$
    – cinch
    Commented Mar 12 at 21:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .