How does noise samples from uniform distribution contribute to the diversity of generator output?

Question

In a Generative Adversarial Network (GAN), there are two multi-layer perceptrons. One is the generator network and another is a discriminator network.

The input for the generator network is a noise vector $z$. The input for a discriminator network is either a generated sample $G(z)$ i.e., the output of a generator network or a training sample $x$ for a training dataset.

My doubt is regarding the input of the generator. The noise vector is generally sampled from the standard normal distribution.

$$z \sim \mathcal{N(0, 1)}$$

Although I am not sure, I think : since the values in the normal distribution vary, the output of the generator can vary accordingly.

But some of the research papers say that the noise vector can also be sampled from a uniform distribution i.e., $z \sim \mathcal{U(a, b)}$ for $a<b$.

$$ U(x) = \begin{cases} \dfrac{1}{b-a} & x\in [a, b] \\ 0 & x\not\in [a, b] \\ \end{cases} $$

It is clear that uniform distribution does not vary like normal distribution and takes only two possible values, hence all samples have equal probability in the given range. Then how can it contribute to the diversity of the output of the generator network?

Answer 1

As pointed out in the comments, a random variable $z$ sampled from a uniform distribution $\mathcal{U}(a,b)$, doesn't take only two values but any value between $a$ and $b$ with equal probability $\frac{1}{b-a}$.

The distribution you use for $z$ in a GAN doesn't have a theoretical justification. The fact that it's an $\mathcal{N}(0,1)$ is never used in the proof of the convergence of the generator to the data distribution in the original GAN paper (see Section 4.1).

However, it should be easy and fast to sample from given that you must take samples at each training step. Why certain authors choose one distribution over the other is usually down to empirical trial and error. GANs are notoriously hard to train and the distribution of $z$ is one of the many hyperparameters of the model.

Other generative models though have theoretical justification for the choice of the distribution of the random variable given to the decoder. In VAEs, it is a Gaussian $\mathcal{N}(0,1)$ because the loss function contains a KL term $\text{KL}(p_\theta(z|x)||p(z))$ which has a nice closed form when $p_\theta(z|x)$ and $p(z)$ are Gaussian distributions (see Section 3 in the original VAE paper).

Answer 2

The main idea of the latent vector $z$ is that it represents some high-level features of images, and the interpretation could be that latent features are normally distributed.

However, since we are trying to approximate (map) a complex multimodal real distribution using a simple one, the feature space $Z$ becomes entangled. So, in practice, we just feed random noise into the generator to add stochasticity. Otherwise, the network would produce the same output each time.

As already discussed in the comments, it doesn't really matter which distribution you use. The only advantage I see of using a normal distribution is that we can use the truncation trick to get better (but less varied) results.