0
$\begingroup$

For a variational autoencoder, we have input $x$ (assume 1 data point for now, like an image), a latent code sampled from the decoder, $z$, and an output $\hat{x}$.

If I were to draw a diagram for the VAE with the input, output, and latent code sample, is it appropriate to write those three as random variables/vectors? Or as instances of random variables/vectors?

I thought it was random variables/vectors, but I saw this discussion, where they talk about the dataset being instances.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

The VAE attempts to model a specific probabilistic (directed) graphical model (Bayesian network)

enter image description here

So, in this PGM, $\mathbf{z}$ and $\mathbf{x}$ are random variables. In principle, I think you could also model $\phi$ and $\theta$ as random variables (in Bayesian statistics, you can also model parameters as random variables and put priors on them).

In practice, the VAE attempts to learn a generative model given a dataset. In fact, the technical part of the VAE paper (section 2.1) starts with

Let us consider some dataset $\mathbf{X}=\left\{\mathbf{x}^{(i)}\right\}_{i=1}^{N}$ consisting of $N$ i.i.d. samples of some continuous or discrete variable $\mathbf{x}$. We assume that the data are generated by some random process, involving an unobserved continuous random variable $\mathbf{z}$.

Later, they use this dataset to define the likelihood

$$\log p_{\theta}\left(\mathbf{x}^{(i)}\right)=D_{K L}\left(q_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\theta}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)\right)+\mathcal{L}\left(\boldsymbol{\theta}, \boldsymbol{\phi} ; \mathbf{x}^{(i)}\right)$$

and, consequently, also the objective function (the Evidence Lower BOund, aka ELBO).

$$ \mathcal{L}\left(\boldsymbol{\theta}, \phi ; \mathbf{x}^{(i)}\right)=-D_{K L}\left(q_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right) \| p_{\theta}(\mathbf{z})\right)+\mathbb{E}_{q_{\phi}\left(\mathbf{z} \mid \mathbf{x}^{(i)}\right)}\left[\log p_{\theta}\left(\mathbf{x}^{(i)} \mid \mathbf{z}\right)\right] $$

Note that we use $\mathbf{x}^{(i)}$ in the formulas above, so the distributions are conditioned on the given samples/dataset and the likelihood function is defined in terms of the samples (that's what a likelihood usually is: a function of the parameters given the usually fixed data).

In the other more extensive paper about VAEs (by the same authors of the VAE), you also have a diagram of the VAE.

enter image description here

So, I think an answer to your question depends on what you actually want to show with the diagram. If you want to show how VAEs are trained, then you definitely need to show that we have a dataset. If your diagram is supposed to show the distributions that the VAE attempts to model, then you can probably use a PGM.

Improve this answer
$\endgroup$
3
  • $\begingroup$ So like when we are sampling from the encoder, we use the reparameterization trick, $\epsilon \sim p(\epsilon)$, $z=\mu+ z\odot \sigma$, and then input $z$ into the decoder. As far as I know, when we are sampling from a distribution, like $\epsilon \sim p(\epsilon)$, $\epsilon$ is a random variable/vector, and then . Is there some point along the way that we start treating it (either $\epsilon$ or $z$) as an instance instead of a random variable/vector? $\endgroup$
    – a12345
    Dec 17, 2021 at 2:35
  • 1
    $\begingroup$ @a12345 Before you sample, you can think that there's a random variable $\epsilon$ that follows a distribution $p$. Once you sample $z$ using the re-parametrization trick, you can think of it as a real data sample/instance. I think that people talk about random variables a little bit imprecisely in machine learning. Sometimes, they might say that $\epsilon$ is a random variable because it was randomly sampled from a distribution. What we can say is that, once we sample $z$, which we pass to the decoder, $z$ is no more a random variable, it's a (random) sample, i.a. sample chosen randomly. $\endgroup$
    – nbro
    Dec 17, 2021 at 8:59
  • 1
    $\begingroup$ Note that, in my comment above, I used the letter $\epsilon$ once to refer to a random variable and other times to refer to a sample from that probability distribution associated with that random variable. $\endgroup$
    – nbro
    Dec 17, 2021 at 9:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .