So, with VAE we use ELBO instead of directly maximizing the marginal likelihood, because the marginal likelihood is intractable.

As far as I understand it, this is the case for two reasons:

  1. $$p(x) = \int p(x,z) \, dz = \int p(x|z;\theta) \, p(z) \, dz$$
  2. Since we sample our latent variables directly from the prior p(z), which is often given analytically in the form of a simple distribution, we're not bound to sample anything meaningful. That is, if there is a high density in an area $$Z_i$$ it is not bound to have any meaningful connection to, how often its corresponding variable appears in our dataset. Assuming our dataset represents "meaningful" examples, we're essentially sampling and maxing our likelihoods with samples, that carry barely any information and can therefore not help us train the model.

Now, my question is; Isn't this only the case at the beginning of the training process? Or is it only with the VAE, because of the regularization term, that the p(z) begins to carry meaningful information?

What I am trying to say is; At some point, say if we have a pretrained decoder on the same dataset, will the samples that have a high probability in p(z) not also carry some information? Will our decoder not have mapped the relation between those and the images, such that it "forces," those z's which appear often in our sample to represent the images that appear often?

Kind of like how the regularization term in the ELBO loss function forces the posterior to map a relationship between the images that appear often to high probability densities in the Z-space?

So, if we let it run for infinity, we could still theoretically learn this way? I get that it would still be intractable, but this would still help me better understand how VAEs work.



You must log in to answer this question.