If the Generator in a GAN is taking a matrix of size WxH of noise to generate a WxH sized output image, and the Discriminator classifies the output as fake, how is that information back-propagated through the generator?

How is the error in classification attributed to individual "pixels" of the generators generated image? Is the error divided by the number of pixels?


2 Answers 2


If you ask this question it means you conceive a generative adversarial network as a combination of 2 separate entities, the discriminator and generator, but this is not really the case.

It is true that for convenience we distinguish between generator and discriminator since they fulfill separate purposes, but by simply looking at a drawing of a whole GAN you'll see that they are not separated at all.

When training the generator we simply backpropagate the gradients coming from the discriminator through the fake generated sample, as it was an intermediate layer between generator and discriminator (see red area in the drawing).

Of course the discriminator can be updated on its own, and even the generator since we can compute loss and backpropagate from the fake generated sample level. This is nothing special, for example it's done also in normal CNN training when using losses computed at the feature maps level (like perceptual loss).

enter image description here

  • $\begingroup$ Maybe reference the separate D/G training steps for detail at the end, as you cover how the architecture supports this. I suspect the OP would appreciate a reference to the full algoirithm. $\endgroup$ Mar 16, 2022 at 11:54
  • $\begingroup$ Thanks for the answer @edoardo, but to confirm, the backprop stage goes from the discriminators output back through to its inputs (the generated image) and continues backwards though the generator? So each pixel the generator output is being corrected by the chain-rule back through the discriminator? $\endgroup$ Mar 16, 2022 at 21:37
  • $\begingroup$ @NeomerArcana yes, exactly! And this doesn't require any fancy implementation when using deep learning libraries, as long as both, discriminator and generator are implemented as modules, autograd automatically recognize that the fake image has gradients coming from the generator, so when backpropagating the discriminator loss you're already updating the generator as well. The only difference required is a different optimizer for the generator weights. $\endgroup$ Mar 17, 2022 at 7:34

I'm going to keep my answer relatively high-level and avoid details like the actual loss functions or activation functions. But please know these also have an effect on GANs.

The Discriminator (D)

The discriminator in a GAN is a binary classifier. It is given an image and asked to predict whether that image is real or fake. The discriminator reduces the given image down using convolutional layers. Convolutional layers detect features, in the case of images these would be edges, borders, certain shapes and so on.

The discriminator is learning which features correspond to a real image and which features correspond to a fake image.

The Generator (G)

The generator is almost the inverse of the discriminator: Instead of reducing features it uses deconvolutional layers to create features from a random seed. This random seed is a vector (not a matrix) called the latent vector (z).

The generator is learning which features to create to return a real label from the discriminator.

How GAN learns

In each training step the following happens:

  1. Get gen_images from G given z.
  2. Get real_predictions from D by passing real images to D.
  3. Get fake_predictions from D by passing gen_images to D.
  4. Compute loss on G as a function of (real, fake_predictions)
  5. Compute loss on D as a function of (real, real_predictions) + (fake, fake_predictions)
  6. Use backpropagation to update the weights and biases in D and G for the losses.

The big point here is that the generator's loss function directly depends on the output of D. Did G manage to trick D or not? This is computed by comparing the fake_predictions with the real label. G wants those two values to be as close as possible.

This loss does not correspond to individual pixels, but because we are using convolutions/deconvolutions in our models it corresponds to feature selection (in D) and feature creation (in G) with groups of pixels.

In short: the individual pixels are not directly trained in a GAN's generator, instead patterns and features creation are.

If you are still confused: Think about how a child draws.

A child doesn't start drawing by examining things a millimeter at a time, but by using symbolism and feature selection. At the beginning you can really only guess what they are trying to make.

A cat might be a big mess of scribbles, but slowly the child learns to draw pointy ears, or whiskers. At some point the child can draw features that are identifiable as a cat. The child has gotten feedback from you when you ask her, "Oh, what's that supposed to be?" and when you say, "Oh, is that a cat?" and then finally, "That's a nice looking cat!".

  • $\begingroup$ How does this work in practice? If the discriminator predicts correctly that the image is a fake, aren't the outputs of the generator (the image pixels) corrected? $\endgroup$ Mar 16, 2022 at 21:51
  • $\begingroup$ Yes! That's what the loss function (for the generator) does. The loss function quantifies how undesired the output was. Since G wants to trick D, we compare the guesses of D (the actual output) on the generated images and compare the guesses to the label real (the desired output). Like in any other network, when the desired output and the actual output don't match there is a loss, which is backpropagated through the network. $\endgroup$ Mar 17, 2022 at 14:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .