From what I understand, a Generative Adversarial Network (GAN) is composed of an encoder (generator), some synthetic data (fake data) and a discriminator that will penalize any distinguishable real data from the fake ones. This will result in 2 training phases where one tries to beat the other one and vice versa.
This force the generator to mimic a distribution given by the synthetic data, thus creating a latent space similar the fake data.
However, in Variational Autoencoders (VAE), the usual loss function also includes a penalty in the latent space when the data do not follow a normal distribution. From what I've learned, this also strongly encourages the latent space to follow a normal distribution.
My question is how does the penalty in the VAE is different from training a whole new network, the discriminator, that in the end penalize the generator for not following the synthetic data distribution?
More generally, is it similar to penalize a wrong latent space distribution directly in the model's loss function than it is with a discriminator network?
Maybe it's easier with GAN to make new distributions, at the cost of training another network, than it is to model an adapted loss function. (?)