Why does the critic of WGAN-GP run more steps than the generator?

Question

As noted in the paper introducing WGAN-GP (see the pseudo-code), for each minibatch of data, the generator's weights are updated only once, and the critic (or discriminator) is updated multiple times. This is also evident from the Keras implementation. However, normal GANs update both the generator and the discriminator only once per batch, as is demonstrated in the TensorFlow implementation. Why this difference?

Answer 1

Balancing the training of the generator and the critic is essential for high-quality GANs. If either the generator or the discriminator overpowers the other, the model will not converge to a satisfactory state.

Balancing the generator and the discriminator can be done in many different ways. For example, you could make the generator bigger, giving it more computational capacity, but potentially slowing down its convergence because there are more parameters to tune.

The way you state, to update either the generator or the discriminator more than the other, is another way of trying to achieve such a balance. Although some tutorials of normal GANs might not show that this balancing is necessary (as they try to make the tutorial as easy as possible), balancing a traditional GAN by training one module more than the other is very common.

With WGAN-GP, we penalize the discriminator which hence learns less fast. As a result, the assumption is that we need to train it more to stay in balance. Consequently, the paper proposes the balancing hyperparameter that it does.

Concluding, this balancing parameter can also be added to normal GANs, or any GAN for that matter. The paper proposes a hyperparameter setting indicative for their test results. However, please play around with it if you want to apply it for your specific application. I've trained GANs both with training the generator 10x more than the discriminator and also the other way around. There is no golden rule for this kind of stuff.