Questions tagged [variational-autoencoder]

For questions related to variational auto-encoders (VAEs). The first VAE was proposed in "Auto-Encoding Variational Bayes" (2013) by Diederik P. Kingma and Max Welling. There are several other VAEs, for example, the conditional VAE.

Filter by
Sorted by
Tagged with
1
vote
0answers
13 views

Compressing Parameters of an Response System

So I have an input-output system, which is fully determined by 256 parameters, of which I know a significant amount are of less importance to the input-output pattern. The data I have is some (64k in ...
1
vote
0answers
16 views

Why would a VAE train much better with batch sizes closer to 1 over batch size of 100+?

I've been training a VAE to reconstruct human names and when I train it on a batch size of 100+ after about 5 hours of training it tends to just output the same thing regardless of the input and I'm ...
1
vote
1answer
35 views

How does implementation of VAE's objective function equate to ELBO?

For a lot of VAE implementations I've seen in code, it's not really obvious to me how it equates to ELBO. $$L(X)=H(Q)-H(Q:P(X,Z))=\sum_ZQ(Z)logP(Z,X)-\sum_ZQ(Z)log(Q(Z))$$ The above is the definition ...
1
vote
0answers
26 views

How can I reconstruct sparse one-hot encodings using an RBM?

I am currently working with a categorical-binary RBM, where there are 50 categorical visible units and 25 binary hidden units. The categorical visible units are expressed in one-hot encoding format, ...
2
votes
1answer
46 views

What is the impact of scaling the KL divergence and reconstruction loss in the VAE objective function?

Variational autoencoders have two components in their loss function. The first component is the reconstruction loss, which for image data, is the pixel-wise difference between the input image and ...
1
vote
1answer
54 views

How many types of variational auto-encoders are there?

I have been studying about auto-encoders and variational auto-encoders. I would like to know how many variants of VAEs are there today. If there are many variants, can they be used for feature ...
0
votes
1answer
29 views

How to construct input dependent convolutional filter?

I am constructing a convolutional variational autoencoder for images, starting out with mnist digits. Typically I would specify convolutional layers in the following way: ...
2
votes
1answer
78 views

What does the notation $\mathcal{N}(z; \mu, \sigma)$ stand for in statistics?

I know that the notation $\mathcal{N}(\mu, \sigma)$ stands for a normal distribution. But I'm reading the book "An Introduction to Variational Autoencoders" and in it, there is this notation:...
0
votes
0answers
39 views

VAE KL divergence loss decreases really fast

I am new to training VAEs and I am using it on some 16x16 images, that contains some images from a physics experiment with one or 2 events i.e. the images are mainly black, except for one or 2 regions ...
0
votes
0answers
108 views

Conditional Variational Autoencoder - NON Image Data

First I would like to expand an issue I've been dealing with way too long: Creating a conditional Variational Autoencoder with continuous variables in non-image data ( more specifically, time series). ...
1
vote
1answer
34 views

What is the main contribution of the paper Disentangling by Factorising?

Considering the paper Disentangling by Factorising, in addition to introducing a new model for Disentangled Representation Learning, FactorVAE (see figure), what is the main theoretical contribution ...
2
votes
0answers
15 views

Does bottleneck size matter in Disentangled Variational Autoencoders?

I suppose that picking an appropriate size for the bottleneck in Autoencoders is neither a trivial nor an intuitive task. After watching this video about VAEs, I've been wondering: Do disentangled ...
1
vote
1answer
65 views

Can a variational auto-encoder learn images composed of random noise at each pixel (each drawn from the same distribution)?

Can a variational auto-encoder (VAE) learn images whose pixels have been generated from a Gaussian distribution (e.g. $N(0, 1)$), i.e. each pixel is a sample from $N(0, 1)$? My gut feeling says no, ...
3
votes
1answer
98 views

Why does the variational auto-encoder use the reconstruction loss?

VAE is trained to reduce the following two losses. KL divergence between inferred latent distribution and Gaussian. the reconstruction loss I understand that the first one regularizes VAE to get ...
1
vote
1answer
120 views

Why do we regularize the variational autoencoder with a normal distribution?

When we define the loss function of a variational autoencoder (VAE), we add the Kullback-Leibler divergence between the sample taken according to a normal distribution of parameters: $$ N(\mu,\sigma)...
2
votes
1answer
85 views

Why does KL divergence not satisfy the triangle inequality?

$D_{KL}=\sum_i p(x_i)log(p(x_i)/q(x_i)$ Also can't you make it satisfy the triangle inequality by taking the absolute value of the information at every point?
4
votes
0answers
101 views

Why can't VAE do sequence to sequence name generation?

I'm working on research in this sector where my supervisor wants to do cannonicalization of name data using VAEs, but I don't think it's possible to do, but I don't know explicitly how to show it ...
1
vote
1answer
39 views

In this VAE formula, why do $p$ and $q$ have the same parameters?

In $$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i),$$ why does $p(x^1,...,x^N)$ and $q(z|x^i)$ have the same parameter $\theta?$ Given that $p$ is ...
2
votes
1answer
112 views

Why is the evidence equal to the KL divergence plus the loss?

Why is the equation $$\log p_{\theta}(x^1,...,x^N)=D_{KL}(q_{\theta}(z|x^i)||p_{\phi}(z|x^i))+\mathbb{L}(\phi,\theta;x^i)$$ true, where $x^i$ are data points and $z$ are latent variables? I was ...
1
vote
1answer
55 views

Why is exp used in encoder of VAE instead of using the value of standard deviation alone?

There's one VAE example here: https://towardsdatascience.com/teaching-a-variational-autoencoder-vae-to-draw-mnist-characters-978675c95776. And the source code of encoder can be found at the ...
2
votes
1answer
42 views

What is the mean in the variational auto-encoder?

Here's a diagram of a variational auto-encoder. There are 2 nodes before the sample (encoding vector). One is the mean, one is the standard deviation. The mean one is confusing. Is it the mean of ...
1
vote
0answers
60 views

Is Gradient Descent algorithm a part of Calculus of Variations?

As in https://en.wikipedia.org/wiki/Calculus_of_variations The calculus of variations is a field of mathematical analysis that uses variations, which are small changes in functions and ...
1
vote
0answers
32 views

What's the difference between semi-supervised VAEs and conditional VAEs?

Can someone explain the difference? I'm assuming the difference is just that the neural nets representing the encoder and decoder are trained in a semi-supervised manner in semi-supervised VAE, which ...
1
vote
0answers
50 views

Why is my variational auto-encoder generating random noise?

This is my first variational autoencoder. Background info: I am using the MNIST digits dataset. The model is created and trained in PyTorch. The model is able to get a reasonably low loss, but the ...
2
votes
1answer
170 views

What is the input for the prior model of VQ-VAE?

I'm trying to implement the VQ-VAE model. In there, a continuous variable $x$ is encoded in an array $z$ of discrete latent variables $z_i$ that are mapped each to an embedding vector $e_i$. These ...
2
votes
0answers
42 views

Why does the ELBO come to a steady state and the latent space shrinks?

I'm trying to train a VAE using a graph dataset. However, my latent space shrinks epoch by epoch. Meanwhile, my ELBO plot comes to a steady state after a few epochs. I tried to play around with ...
2
votes
2answers
159 views

What makes GAN or VAE better at image generation than NN that directly maps inputs to images

Say a simple neural network's input is a collection of tags (encoded in some way), and the output is an image that corresponds to those tags. Say this network consists of some dense layers and some ...
3
votes
0answers
38 views

In variational auto-encoders, what should the dimension of the mean and variance of the latent distribution and the latent vector be?

I'm having a problem to understand the needed dimensions of an VAE, especially for mu, logvar and z layer. Let's say I have an input of 512x512, 1 color channel (CT images), batch size 32. Then my ...
5
votes
1answer
70 views

Concrete example of latent variables and observables plugged into the Bayes' rule

In the context of the variational auto-encoder, can someone give me a concrete example of the application of the Bayes' rule $$p_{\theta}(z|x)=\frac{p_{\theta}(x|z)p(z)}{p(x)}$$ for a given latent ...
4
votes
1answer
148 views

Does MMD-VAE solve the problem of blurred images of vanilla VAEs?

I understand that with vanilla VAEs, there are a few reasons justifying the production of blurred out images. The InfoVAE paper describes the case when the decoder is flexible enough to ignore the ...
1
vote
0answers
28 views

Reduce same sample distance in VAE encodings

I'm working on a beta VAE model learning a latent representation used as a similarity metric for image registration. One of the main problems I'm facing is that the encoder + sampler output doesn't ...
2
votes
0answers
44 views

Are there any general tips for troubleshooting a VAE when apparently it is not learning?

I am trying to train a VAE for anomaly detection. I chose one architecture from this Github repository and I adjusted the input and output to match what I need. In my case, the input (and hence the ...
2
votes
1answer
65 views

How does the Kullback-Leibler divergence give “knowledge gained”?

I'm reading about the KL divergence on Wikipedia. I don't understand how the equation gives "information gained" as it says in the "Interpretations" section Expressed in the ...
3
votes
2answers
211 views

What's going on in the equation of the variational lower bound?

I don't really understand what this equation is saying or what the purpose of the ELBO is. How does it help us find the true posterior distribution?
4
votes
1answer
266 views

Is there a continuous conditional variational auto-encoder?

The Conditional Variational Autoencoder (CVAE), introduced in the paper Learning Structured Output Representation using Deep Conditional Generative Models (2015), is an extension of Variational ...
7
votes
1answer
487 views

How should we choose the dimensions of the encoding layer in auto-encoders?

How should we choose the dimensions of the encoding layer in auto-encoders?
3
votes
1answer
2k views

Why doesn't VAE suffer mode collapse?

Mode collapse is a common problem faced by GANs. I am curious why doesn't VAE suffer mode collapse?