I'm working on a variational autoencoder (VAE) with 20 different classes in my training data. I've successfully trained the VAE and can sample from the latent space to generate data points. However, I have a specific requirement: I want to generate samples from the latent space that somehow combines the properties of all 20 classes, rather than just falling somewhere in between two classes.

In other words, I want to generate output images that exhibit features and characteristics that are representative of all 20 classes simultaneously. Is this possible with a VAE, and if so, how can I modify the sampling process or the latent space to achieve this goal? Many of the approaches I see (for example in MNIST) shows the morphing from one digit to another, but this is only between 2 classes.

I understand that the VAE is designed to generate data points that are similar to the training data, but I'm interested in exploring ways to synthesize data that embodies a blend of characteristics from multiple classes. Any insights or guidance on how to approach this task would be greatly appreciated.

  • $\begingroup$ I want to generate output images that exhibit features and characteristics that are representative of all 20 classes simultaneously can you give an example with MNIST? $\endgroup$
    – Alberto
    Commented Sep 4, 2023 at 11:55
  • $\begingroup$ @Alberto doesn't have to look like a real digit, it just has to somehow look like a mix between all (Im working with the omniglot so there are more classes and variation) $\endgroup$ Commented Sep 5, 2023 at 12:31
  • $\begingroup$ Never met in my carrier a method that can give you out of distribution samples out fo the box, if you find any, ping me here, I'm very interested $\endgroup$
    – Alberto
    Commented Sep 6, 2023 at 0:00

1 Answer 1



The problem you're describing is the inability to find a generated image from your latent space that represents an image which interpolates all your different classes. The reality is that a Variational Autoencoder (VAE) cannot guarantee the generation of such an image. VAEs operate by attempting to map a distribution to an image, using parameter vectors from a multidimensional Normal Distribution. The key thing is that the Loss function depends on two different components: the Reconstruction Loss and the KL Divergence. The Reconstruction Loss is the part that tries to imitate the input data, and the KL divergence is the part that tries to converge the distribution to a Normal Distribution with Mean 0 and Standard Deviation 1. Meaning that the model really doesn't care about the way it organizes the latent space, as long as it can minimize both of these conditions.

Center of mass:

A fundamental approach is to find the center of mass of all the classes you wish to combine. Start by identifying the clusters and then employ a k-clustering technique to determine each cluster's central point. Next, locate the center of mass of the selected points with the following formula: $$ \bar{z} = \frac{1}{n} \sum_{i = 1}^{n} z_i $$ where $z_i$ is the ith vector from the center of mass we previously discussed. This will yield the mean point between the classes, possibly aiding in discovering the desired generations.

One challenge with interpolating between these clusters is that you might inadvertently end up in a different cluster. For example, if you aim to interpolate between numbers 1 and 9, you might unintentionally generate a 7 due to the proximity of 1 to the shape of 7. This wouldn't necessarily give you a blend of 1 and 9. You might consider interpreting your latent space using dimensionality reduction techniques like PCA and plotting the results. This could provide insight into how the latent space has clustered the classes. Esentially this is what studies the disentanglement representation theory, a field of study that tries to separate these clusters individually such that we could have more freedom in the generation of our data.

However, it's essential to understand that even after undertaking these steps, there's no assurance that the outcome will be a blend of all your classes.

How to solve the problem

So most of the time, if you don't apply any type of inductive bias both to the model and the dataset, you will end up with an entangled latent space. This means that the data classes are not well separated from each other. This is why there exist different methods around the VAE, although I'm not an expert in the field. I can share with you a paper about it. These are maybe the reason why H-VAE, VQ-VAE or other types of VAEs are able to get more realistic and diverse results;because of their disentangled representation.

I did some experiments with different types of VAEs that use some type of inductive bias to provoke cluster separation. This final result isn't as crisp as I would like but I will share with you the code here I used to generate this images if you want to generate your own results.

enter image description here

"From these visualizations, it can be seen that the different inductive biases implemented help separate the classes in the latent space."

Eigen Decompositions:

I'd also like to introduce you to an alternative technique that doesn't rely on VAEs but potentially offers more accurate results for the things you are searching for: Eigenfaces. The term "Eigenfaces" originates from its initial application on face datasets. The concept involves utilizing the Singular Value Decomposition (SVD) of your entire dataset and visualizing the principal values or eigenvectors of the dataset matrix. Unlike VAEs, SVD identifies the component images that are crucial for minimizing the variance of the image when combined linearly. Here's an example I generated using the MNIST dataset, showcasing the first eight component images that offer a compelling mean representation of the entire dataset.Link to Snippet.

enter image description


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .