Is plain autoencoder a generative model?

Question

I am wondering how a plain auto encoder is a generative model though its version might be but how can a plain auto encoder can be generative. I know that Vaes which is a version of the autoencoder is generative as it generates distribution for latent variables and whole data explicitly. But I am not able to think how an autoencoder generates probability distribution and becomes a generative model.

Also from this youtube video: here It says plain auto encoder is not a generative model. See last line from picture.

Answer 1

An autoencoder is not considered a generative model, because it only reconstructs the given input. You could use the decoder like a generative model by putting in different vectors. However, the standard autoencoder mostly learns a sparse latent space. This means that you will have distinct clusters in the latent space (see the left image below). The decoder has never learned to reconstruct vectors in between the clusters, so it will produce very abstract things - mostly garbage.

Instead a variational autoencoder (VAE) is considered a generative model. It's basically an autoencoder with a modified bottleneck. This VAE learns a dense latent space (see image on the right), this means you can sample any vector from the latent space, pass it to the model and it will give you a nice result with somewhat interpolated object properties from the dataset.

This article provides a nice overview of the two models.

Figure taken from here

Answer 2

Just as completing, in general, the autoencoders are an unsupervised learning technique in which we use neural networks for the task of representation learning. Specifically, we'll design a neural network architecture to impose a network bottleneck, forcing a compressed knowledge representation of the original input. In this case, we dont generate any new data and just compress them. In the case of VQ-VAE or VAE, we generate new data because the architecture encodes all data in latent space and samples from that (continuous in VAE and discrete in VQ-VAE). Therefore, although the decoder will generate similar outputs (if regulated well), the input data is not just a compressed version.For better understanding, in VAE, the mean and variance of encoded data are used for generation.

Also, you can read these good references: https://www.jeremyjordan.me/autoencoders/ , https://www.v7labs.com/blog/autoencoders-guide , https://www.analyticsvidhya.com/blog/2021/06/complete-guide-on-how-to-use-autoencoders-in-python/