2
$\begingroup$

I am trying to reason to myself why is it that VAEs can approximate arbitrary probability distributions even though 𝑞𝜙(𝑧|𝑥) and 𝑝𝜃(𝑥|𝑧) are Gaussian.

I understand that the parameters are typically learned using neural networks 𝜙=𝑓𝑤(𝑥) and 𝜃=𝑔𝑤(𝑧) , where 𝑓 and 𝑔 denote arbitrary neural networks. Therefore, some ideas I had is perhaps the entire VAE is able to learn differentiable, change of variables mappings, which allows the VAE to ensure that, although 𝜙 and 𝜃 both characterize Gaussian distributions, they do so using some transformation of the original variables 𝑥 and/or 𝑧 and so effectively correspond to some arbitrary distribution rather than a Gaussian in 𝑥 or 𝑧 .

However, and I may just be unnecessarily picky here, but I can't quite convince myself why the Gaussian PDF itself, even with the change of variables mapping, will not act as a "bottleneck" in learning some rather complex non-Gaussian distributions. I was wondering if there is some universal approximation theorem in the case of VAEs, as there in feed-forward neural networks / RNNs etc.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The fact that you can approximate any distribution is given by the definition ELBO, which is a lower bound in order to learn $p(x)$

Theoretically speaking, if you are able to make that difference between $p(x)$ and ELBO $0$, than you can say that you have actually approximated $p(x)$ correctly

This usually doesn't happen, as it is an ideal/optimal solution, which is not what happens when you then introduce mathematical models, optimized with gradient descent, and moreover non convex

However, you might want to take a look at optimal transport, so you might get a better idea how to transform an arbitrary complex distribution, to a simple one

About the bottleneck, you can think that your NN is so powerful, that might learn a transformation between your more-complex-prior to a Gaussian, thus a bigger NN might learn the transformation $p_\text{data}\rightarrow p_\text{complex prior} \rightarrow p_\text{Gaussian}$

Improve this answer
$\endgroup$
5
  • $\begingroup$ I see, thank you for your fast response. So are you essentially saying that the neural network may learn how to "morph" the space of $x$ using a differentiable mapping into some Gaussian distribution that it can then leverage from that point onward? This was my attempt at summarizing your last point. Edit: thanks also for pointing me towards optimal transport. $\endgroup$
    – Decaying Tails
    Commented Aug 7, 2023 at 22:47
  • $\begingroup$ @Joel yes, you can phrase it like that... you can even think that each layer learns a step towards that simpler distribution... however, OT transforms distributions, NN transform a sample of that distribution $\endgroup$
    – Alberto
    Commented Aug 8, 2023 at 8:38
  • $\begingroup$ I see @Alberto Sinigaglia, thank you. I am wondering, however, if the mapping of x to the parameterization of the latent space (usually Gaussian) is deterministic, then why can't we view the process as transforming the entire distribution itself? $\endgroup$
    – Decaying Tails
    Commented Aug 10, 2023 at 0:28
  • $\begingroup$ On second thoughts, I think you mean that the parameters of $p(z|x)$ itself are usually dependent on $x$ and so the NN is technically mapping the sample $x$ to a different distribution - am I interpreting you correctly? $\endgroup$
    – Decaying Tails
    Commented Aug 10, 2023 at 0:35
  • $\begingroup$ @Joel well, first, transforming samples to look like a distribution (VAE), it's much simpler that transforming in closed form the whole distribution (Opt Transport)... secondly, the fact that we are using a distribution in the latent space, it's not always necessary... you can achieve a pseudo-VAE with just a naive AE with a L2 regularization on the latent vectors $\endgroup$
    – Alberto
    Commented Aug 10, 2023 at 11:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .