Explain the difference in graphical patterns between discriminator fake loss and generator loss in GAN

Question

In GAN (generative adversarial networks), let us take "binary cross-entropy" as the loss function for discriminator $$(overall \; loss = -\sum log(D(x_i)) -\sum log(1-D(G(z_i))) $$ $$ where \; x_i = real \; image \; pixel \; matrix$$ $$ and \; z_i = a \; vector \; from \; latent \; space$$. Let us define discriminator real loss and fake loss: $$ d_{fake \; loss} = -\sum log(1-D(G(Z)))$$ $$ d_{real \; loss} = -\sum log(D(x))$$ $$ d_{fake \; loss} \; implies \; discriminator \; loss \; against \; fake \; images$$ $$ d_{real \; loss} \; implies \; discriminator \; loss \; against \; real \; images$$ Generator Loss : $$ g_{loss} = -\sum log(D(G(z_i)))$$ Since the functions are similar, we should be expecting some similarity in graphical patterns (i.e since none of the functions are inherently oscillatory, I expect that if one comes out to be oscillatory, the other one should be the same as well). But, If you refer to chapter 10 of the book "Generative Adversarial Networks with python by Jason Brownlee", we find some difference. The following are the graphs published in the book

Can anyone explain the difference in the plots between discriminator fake loss and generator loss (mathematically)?

Answer 1

Your statement that we should be expecting some similarity in graphical patterns is not correct. The GAN loss takes the following form:

$$ \min_G \max_D V(D,G) = \mathbb{E}_{x \sim p_{data}(x)} [\log D(x)] + \\ + \mathbb{E}_{z \sim p_z(z)} [\log (1 - D(G(z)))]\label{minimax} $$

We want to maximize this loss w.r.t. D in order to distinguish between real and fake samples ($D(x)\rightarrow 1$ and $D(G(z))\rightarrow 0$), whereas the task of G is exactly the opposite. We want to minimize the function w.r.t. G so that the difference between real and generated data will be minimal. Thus, the problem becomes a minimax non-cooperative game. Here is a good explanation of the loss.

Training GANs requires finding a Nash equilibrium of a non-convex game with continuous, high dimensional parameters. GANs are typically trained using gradient descent techniques that are designed to find a low value of a cost function, rather than to find the Nash equilibrium of a game. Two models are trained simultaneously and each model updates its cost independently with no respect to another player in the game.

In other words, as soon as D becomes better, G becomes also better. Updating the gradient of both models concurrently cannot guarantee a convergence. For more information, see Improved Techniques for Training GANs and Wasserstein GAN.

Moreover, since D and G play a non-cooperative game, it can be shown that there are some cases when it is impossible to find a Nash equilibrium at all: