13
$\begingroup$

Starting from my own understanding, and scoped to the purpose of image generation, I'm well aware of the major architectural differences:

  • A GAN's generator samples from a relatively low dimensional random variable and produces an image. Then the discriminator takes that image and predicts whether the image belongs to a target distribution or not. Once trained, it can generate a variety of images just by sampling the initial random variable and forwarding through the generator.

  • A VAE's encoder takes an image from a target distribution and compresses it into a low dimensional latent space. Then the decoder's job is to take that latent space representation and reproduce the original image. Once the network is trained, it can generate latent space representations of various images, and interpolate between these before forwarding through the decoder which produces new images.

What I'm more interested is the consequences of said architectural differences. Why would I choose one approach over the other? And why? (for example, if GANs typically produce better quality images, any ideas why that is so? is it true in all cases or just some?)

Improve this question
$\endgroup$
3
  • $\begingroup$ You're already describing both approaches, and, as far as I know, these descriptions are accurate. So, are you asking that someone tries to tell you differences that you are not able to capture from your description? Is this your first question? I suppose, of course, you're also interested in knowing if you missed something in terms of the high-level description of both the VAE and GAN. $\endgroup$
    – nbro
    Commented Jan 6, 2021 at 11:49
  • $\begingroup$ The other question "when would you use one over the other" is actually very interesting and useful, I would say, given that they are both generative models, and it's not clear when one would perform better than the other, so when to choose one over the other. $\endgroup$
    – nbro
    Commented Jan 6, 2021 at 11:50
  • 1
    $\begingroup$ @nbro yes I mean the more interesting question. I hoped using "fundamental" in the title would hint that I don't care about the obvious architectural differences. I'll edit to try and make it clearer. $\endgroup$
    – Alexander Soare
    Commented Jan 6, 2021 at 11:55

1 Answer 1

Reset to default
6
$\begingroup$

GANs generally produce better photo-realistic images but can be difficult to work with. Conversely, VAEs are easier to train but don’t usually give the best results.

I recommend picking VAEs if you don’t have a lot of time to experiment with GANs and photorealism isn’t paramount.

There are exceptions such as Google’s VQ-VAE 2 which can compete with GANs for image quality and realism. There is also VAE-GAN and VQ-VAE-GAN.

As a note, GANs and VAEs are not specifically for images and can be used for other data types/structures.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks Brian. I think this answers the question, although I will also change my question a bit as I realised this is not exactly what I'm looking for. Looking to get something a little more fundamental. $\endgroup$
    – Alexander Soare
    Commented Jan 6, 2021 at 12:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .