1
$\begingroup$

I've been reading about GANs so I can implement a simple image generator. It seems that the loss for the generator is given by the following equation: log(1 - D(G(z)))

But I don't see how this equation can apply to the generator. The generator's output is in latent space (e.g. 784) but the discriminator's output is in label space (e.g. 10). Therefore this loss would have 10 dimensions. So how can the generator, which outputs 784 dimensions, be hooked up to minimise this 10 dimensional function? What am I missing?

$\endgroup$

1 Answer 1

1
$\begingroup$

What am I missing?

The generator's loss is not calculated by comparing its output to a target value, but by processing it through the discriminator. So it is still technically a function of its output, but a complex and indirect one.

There will consequently be a gradient at the generator's output, and you could potentially derive some kind of per-pixel loss metric using that, and perhaps summarise it into a single mean value. However, I don't think that would give you any useful extra information for monitoring progress. The objective function used for training is still "how well can the discriminator tell that the generator's image is not real?".

Either way, progress when training GANs is hard to assess. You don't want to see very low loss values for discriminator or generator because they imply instability. You may be able to use a scoring system like Inception Score, or Fréchet Inception Distance which use a trained image classifier to assess quality of the output independently of generator or discriminator scores.

GANs are not the only neural network architecture and training scheme where the objective function is a complex processing of the output layer. Face recognition using triplet loss also has an indirect function (this time not via a separate network). In both this case, and with GANs, you rely on being able to back-propagate through the objective function to get gradients that affect the network you are training.

The take away is that objective functions do not have to be limited to direct comparisons of a neural network's output with some target value. They can be more complex than that, and this can allow invention of new types of supervised and semi-supervised training feedback.

$\endgroup$
3
  • $\begingroup$ Thanks for your response! It seems you are you saying that the gradient is derived from the discrimnator's loss. So I can almost think of the generator and the discriminator as one big combined network. Is this correct? $\endgroup$
    – zacoons
    Sep 24, 2023 at 21:28
  • $\begingroup$ @Zacoons sort of, yes. The generator gradient is not calculated from discriminator's loss, but it does require processing by the discriminator. It's kind of the opposite of the discriminator loss, although it doesn't evaluate any real images, only the fake ones. Absolutely correct on "one big network", if you add that the discriminator's weights are not updated for generator loss $\endgroup$ Sep 25, 2023 at 7:18
  • $\begingroup$ I think I understand now. I've managed to make a simple implementation and it works somewhat (though that doesn't necessarily mean I'm doing it right). Your help is much appreciated (I'd upvote but I need >15rep). God bless. $\endgroup$
    – zacoons
    Sep 25, 2023 at 22:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .