2
$\begingroup$

I am learning about generative models, and I don't quite understand how the GAN architecture can maintain similar generated images close in the latent space. For example, an autoencoder and a variational autoencoder can map similarly generated images very close in their latent space representation. This can be done because the encoder learns to map similar images in the latent space in order to reduce the loss.

However, in the case of a GAN, there is no encoder. Instead, the latent space comes from a high-dimensional Gaussian distribution. The problem is that each time a vector is sampled from the latent space, it is a completely random vector from the distribution. This could lead to the following possibilities:

First: Very different images could be sampled with very close latent points. This would mean that we could have similar images in different parts of the space. Second: Non-similar images could be very close in their latent representation. Meaning that a single point could represent two different types of images. My problem comes with the question: If a latent point is sampled randomly with each generation, how is it possible to cluster similar images in the latent space?

$\endgroup$

1 Answer 1

1
$\begingroup$

The GAN generator is an encoder from a latent space. The latent space is unconstrained by any individual items of training data, it doesn't matter which real images are shown to help train the discriminator. Training the discriminator to correctly classify "real" images is handled as a separate stage of training to the stage of training on fake images, and there is no direct link between the training images used and the generator's output.

The discriminator does not take the latent space as an input, only an output image to classify as real or fake. As such, the discriminator cannot provide feedback based on the "closeness" of images or not to a target expression of the latent space, only whether it can differentiate between generated and real images. This is very different to a Variational Autoencoder (VAE), which is trained based on reconstruction errors from specific target images.

The generator is therefore free to create an arbitary latent space during training to represent any subset of images that will fool the discriminator. As there is no strong reason to create strongly delineated sub-spaces of output based on the input, the generator will naturally tend to produce similar/related images when the inputs are similar.

The generator doesn't have to produce images from either a smooth or noisy latent space. It will just tend towards what the architecture and initial weights encourage. As it happens, this will more often be a smoothly interpolatable space than a high frequency pseudo-random one.

First: Very different images could be sampled with very close latent points.

This could happen, but only if the generator was already producing very different images from close latent points due to its current weights. There is no association with real training images for the discriminator - which of those images are sampled are completely independent of the generations.

Second: Non-similar images could be very close in their latent representation. Meaning that a single point could represent two different types of images.

A single point won't be required to represent two different images, because it doesn't have to represent anything specific. It is free to drift to whatever the generator makes with that output, under only the constraint that it fools the discriminator. It doesn't have to fool the discriminator that it has made a specific image, but just any image from the class of "real images".

In general, close latent spaces can produce very different images, and this can happen, but there will often be a tendency to create a smooth latent space, because that is an easier set of weights to learn for most neural networks. NNs tend towards global average solutions first during training, and add detail after - this is why early stopping works well as a regularisation technique for neural networks.

In practice, GANs can suffer from the opposite problem - the outputs becoming too similar regardless of input position in the latent space, and lacking the full gamut of allowed values from the training data. This can result in a failure state called mode collapse where the generator focuses output on a small subsection of possible outputs, which the discriminator will decide on average are always fake, and the whole training process become stuck.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .