How is the output of the Generator in a GAN corrected?

Question

If the Generator in a GAN is taking a matrix of size WxH of noise to generate a WxH sized output image, and the Discriminator classifies the output as fake, how is that information back-propagated through the generator?

How is the error in classification attributed to individual "pixels" of the generators generated image? Is the error divided by the number of pixels?

Answer 1

If you ask this question it means you conceive a generative adversarial network as a combination of 2 separate entities, the discriminator and generator, but this is not really the case.

It is true that for convenience we distinguish between generator and discriminator since they fulfill separate purposes, but by simply looking at a drawing of a whole GAN you'll see that they are not separated at all.

When training the generator we simply backpropagate the gradients coming from the discriminator through the fake generated sample, as it was an intermediate layer between generator and discriminator (see red area in the drawing).

Of course the discriminator can be updated on its own, and even the generator since we can compute loss and backpropagate from the fake generated sample level. This is nothing special, for example it's done also in normal CNN training when using losses computed at the feature maps level (like perceptual loss).

Answer 2

I'm going to keep my answer relatively high-level and avoid details like the actual loss functions or activation functions. But please know these also have an effect on GANs.

The Discriminator (D)

The discriminator in a GAN is a binary classifier. It is given an image and asked to predict whether that image is real or fake. The discriminator reduces the given image down using convolutional layers. Convolutional layers detect features, in the case of images these would be edges, borders, certain shapes and so on.

The discriminator is learning which features correspond to a real image and which features correspond to a fake image.

The Generator (G)

The generator is almost the inverse of the discriminator: Instead of reducing features it uses deconvolutional layers to create features from a random seed. This random seed is a vector (not a matrix) called the latent vector (z).

The generator is learning which features to create to return a real label from the discriminator.

How GAN learns

In each training step the following happens:

1. Get gen_images from G given z.
2. Get real_predictions from D by passing real images to D.
3. Get fake_predictions from D by passing gen_images to D.
4. Compute loss on G as a function of (real, fake_predictions)
5. Compute loss on D as a function of (real, real_predictions) + (fake, fake_predictions)
6. Use backpropagation to update the weights and biases in D and G for the losses.

The big point here is that the generator's loss function directly depends on the output of D. Did G manage to trick D or not? This is computed by comparing the fake_predictions with the real label. G wants those two values to be as close as possible.

This loss does not correspond to individual pixels, but because we are using convolutions/deconvolutions in our models it corresponds to feature selection (in D) and feature creation (in G) with groups of pixels.

In short: the individual pixels are not directly trained in a GAN's generator, instead patterns and features creation are.

If you are still confused: Think about how a child draws.

A child doesn't start drawing by examining things a millimeter at a time, but by using symbolism and feature selection. At the beginning you can really only guess what they are trying to make.

A cat might be a big mess of scribbles, but slowly the child learns to draw pointy ears, or whiskers. At some point the child can draw features that are identifiable as a cat. The child has gotten feedback from you when you ask her, "Oh, what's that supposed to be?" and when you say, "Oh, is that a cat?" and then finally, "That's a nice looking cat!".