In GAN (generative adversarial networks), let us take "binary cross-entropy" as the loss function for discriminator $$(overall \; loss = -\sum log(D(x_i)) -\sum log(1-D(G(z_i))) $$ $$ where \; x_i = real \; image \; pixel \; matrix$$ $$ and \; z_i = a \; vector \; from \; latent \; space$$. Let us define discriminator real loss and fake loss: $$ d_{fake \; loss} = -\sum log(1-D(G(Z)))$$ $$ d_{real \; loss} = -\sum log(D(x))$$ $$ d_{fake \; loss} \; implies \; discriminator \; loss \; against \; fake \; images$$ $$ d_{real \; loss} \; implies \; discriminator \; loss \; against \; real \; images$$ Generator Loss : $$ g_{loss} = -\sum log(D(G(z_i)))$$ Since the functions are similar, we should be expecting some similarity in graphical patterns (i.e since none of the functions are inherently oscillatory, I expect that if one comes out to be oscillatory, the other one should be the same as well). But, If you refer to chapter 10 of the book "Generative Adversarial Networks with python by Jason Brownlee", we find some difference. The following are the graphs published in the book Line plots for loss and accuracy for a stable GAN (1st GRAPH plots the losses)

Line plots for loss and accuracy for a stable GAN with mode collapse

Can anyone explain the difference in the plots between discriminator fake loss and generator loss (mathematically)?

  • $\begingroup$ You can use latex on this site. So, please, edit your post to format it with latex. $\endgroup$
    – nbro
    Nov 12 '20 at 10:41

Your statement that we should be expecting some similarity in graphical patterns is not correct. The GAN loss takes the following form:

$$ \min_G \max_D V(D,G) = \mathbb{E}_{x \sim p_{data}(x)} [\log D(x)] + \\ + \mathbb{E}_{z \sim p_z(z)} [\log (1 - D(G(z)))]\label{minimax} $$

We want to maximize this loss w.r.t. D in order to distinguish between real and fake samples ($D(x)\rightarrow 1$ and $D(G(z))\rightarrow 0$), whereas the task of G is exactly the opposite. We want to minimize the function w.r.t. G so that the difference between real and generated data will be minimal. Thus, the problem becomes a minimax non-cooperative game. Here is a good explanation of the loss.

Training GANs requires finding a Nash equilibrium of a non-convex game with continuous, high dimensional parameters. GANs are typically trained using gradient descent techniques that are designed to find a low value of a cost function, rather than to find the Nash equilibrium of a game. Two models are trained simultaneously and each model updates its cost independently with no respect to another player in the game.

In other words, as soon as D becomes better, G becomes also better. Updating the gradient of both models concurrently cannot guarantee a convergence. For more information, see Improved Techniques for Training GANs and Wasserstein GAN.

Moreover, since D and G play a non-cooperative game, it can be shown that there are some cases when it is impossible to find a Nash equilibrium at all:

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.