# Are the authors of the VAE paper writing the PDFs as a function of the random variables?

Usually, I see the conventions:

• discrete random variable is denoted as $$X$$,
• the pmf is written as $$P(X=x)$$ or $$p(X=x)$$ or $$p_{X}(x)$$ or $$p(x)$$, where $$x$$ is an instance of $$X$$
• a continuous random variable is denoted as $$X$$,
• the pdf is denoted as $$f_{X}(x)$$ or $$f(x)$$, where $$x$$ is an instance of $$X$$; sometimes $$p$$ is used here too instead of $$f$$.

However, the VAE paper uses slightly different notation that I'm trying to understand

Let us consider some dataset $$\mathbf{X}=\left\{\mathbf{x}^{(i)}\right\}_{i=1}^{N}$$ consisting of $$N$$ i.i.d. samples of some continuous or discrete variable $$\mathrm{x}$$. We assume that the data are generated by some random process, involving an unobserved continuous random variable $$\mathbf{z}$$. The process consists of two steps: (1) a value $$\mathbf{z}^{(i)}$$ is generated from some prior distribution $$p_{\boldsymbol{\theta}^{*}}(\mathbf{z}) ;(2)$$ a value $$\mathbf{x}^{(i)}$$ is generated from some conditional distribution $$p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$$. We assume that the prior $$p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$$ and likelihood $$p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$$ come from parametric families of distributions $$p_{\boldsymbol{\theta}}(\mathbf{z})$$ and $$p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$$, and that their PDFs are differentiable almost everywhere w.r.t. both $$\boldsymbol{\theta}$$ and $$\mathbf{z}$$. Unfortunately, a lot of this process is hidden from our view: the true parameters $$\theta^{*}$$ as well as the values of the latent variables $$\mathrm{z}^{(i)}$$ are unknown to us.

So I am looking at these:

• $$p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$$
• $$p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$$
• dataset $$\mathbf{X}=\left\{\mathbf{x}^{(i)}\right\}_{i=1}^{N}$$

So I know the subscript for $$\theta$$ denotes those are the parameters for the pdf. It says "discrete variable $$\mathrm{x}$$", "unobserved continuous random variable $$\mathbf{z}$$", and "latent variables $$\mathrm{z}^{(i)}$$". In the top, where I wrote " discrete random variable $$X$$", seems like that's the equivalent of "discrete variable $$\mathrm{x}$$" in this paper.

So, it looks like they're writing the PDFs as a function of the random variables. Is my assumption correct? Because it is different than the typical conventions I see.

edit: looks like his other paper has a notation guide, in the appendix, though it seems like he's conflating both random vector and instances of vector in the notation? https://arxiv.org/pdf/1906.02691.pdf

• no they’re not writing the pdf/pmf as a function of a random variable. The capital X here is just denoting your dataset of realisations of random variables Nov 30, 2021 at 23:04
• Sorry I rewrote my comment, that particular part about p(X) might have made things more confusing. Nov 30, 2021 at 23:17
• I would say that here they’re saying ‘discrete variable’ to denote a realisation of the random variable, so $p(x)$ in this setting is still in line with standard notation. $p(\cdot)$ is a function of $x$ (discrete in this case), it’s just that $x$ are some random realisations of a random variable $X$. Nov 30, 2021 at 23:28
• One other possibility I'm considering is that they could just be using it for both the instance and random variable? Dec 1, 2021 at 0:39

When it comes to notation/terminology, often, people in machine learning are (a bit?) sloppy, which causes a lot of confusion, especially for newcomers to the field or people not very math-savvy. I was also confused about this notation at some point (see my last questions here, which are all about this confusing topic). See also this answer.

In the VAE paper, $$\mathbf{X}$$ is a dataset, as the authors write.

Your confusion also arises because the authors vaguely use the term "probability distribution", rather than pdf or pmf, to refer, for example, to $$p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$$, which thus does not refer to a pdf or pmf. In fact, the authors also write

their PDFs are differentiable almost everywhere w.r.t. both $$\boldsymbol{\theta}$$ and $$\mathbf{z}$$

The $$\mathbf{z}$$ can refer to

1. a random variable, or
2. an input to the function $$p_{\boldsymbol{\theta}^{*}}$$

If it's the first case, then $$p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$$ is the composition of 2 functions (because a rv is also a function).

If it's the second case, then $$p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$$ is the evaluation of $$p_{\boldsymbol{\theta}^{*}}$$ at $$\mathbf{z}$$.

I think the 2nd case is the most likely. In addition, people are being sloppy here and use the notation $$p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$$ (rather than just $$p_{\boldsymbol{\theta}^{*}}$$) to emphasize $$p_{\boldsymbol{\theta}^{*}}$$ is a function of some input variable (not random variable!), which we denote with the letter $$\mathbf{z}$$ to remind ourselves that $$\mathbf{z}$$ is associated with a random variable denoted with the same letter (and maybe also in bold and lowercase).

So, in this case, let's say we denote the random variable associated with $$p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$$ with $$\mathbf{z}$$, then we could refer to this associated prior more explicitly as follows $$p_\mathbf{z}(\mathbf{z})$$ (but that would even be more confusing). It would have been a better idea to use $$\mathbf{Z}$$, but then we may use the upper case letters to denote matrices or sets (like the VAE paper), so we end up with this mess (which is one of the 2 mythical difficult problems well-known in Computer Science, i.e. naming things), which we need to learn to deal with or just ignore.

Conclusion: when I look at $$p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$$, which has been referred to as a probability distribution, I think there's also some associated random variable, which people, in that same context, will probably denote as $$\mathbf{z}$$ or $$\mathbf{Z}$$. There may also be some input variable (not a random variable), which we denote by $$\mathbf{z}$$ or $$z$$. If they are not mentioned, then I just ignore that. I never think that $$p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$$ is the composition of 2 functions (even if that's the case), because that case was never useful in my readings.

• I just noticed in Appendix A of his other paper on VAEs, he goes over the notation. Seems like he's using the same notation for random vectors and instances of random vectors. arxiv.org/pdf/1906.02691.pdf Dec 2, 2021 at 17:58
• and yeah, figuring out which notation to use is a mess. I was thinking italics, using mathcal in latex , and something like this:tex.stackexchange.com/questions/520412/… gives more variety there. Dec 2, 2021 at 18:03

Machine learning papers are often somewhat confused about the distinction between a distribution and its probability density. I would rewrite this

The process consists of two steps: (1) a value $$\mathbf{z}^{(i)}$$ is generated from some prior distribution $$p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$$; (2) a value $$\mathbf{x}^{(i)}$$ is generated from some conditional distribution $$p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$$.

as follows

The process consists of two steps: (1) a value $$\mathbf{z}^{(i)}$$ is generated from some prior distribution. The probability density of selecting $$\mathbf{z}^{(i)}$$ is known and denoted as $$p_{\boldsymbol{\theta}^{*}}(\mathbf{z}^{(i)})$$. (2) a value $$\mathbf{x}^{(i)}$$ is generated from some conditional distribution. The probability density of selecting $$\mathbf{x}^{(i)}$$ given $$\mathbf{z}^{(i)}$$ is known and denoted as $$p_{\boldsymbol{\theta}^{*}}(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i)})$$.

As for the uppercase/lower case notation, this notation is not used in machine learning. $$z, \, x$$ are both random variables. In this paper, the authors use $$z^{(i)}$$, $$x^{(i)}$$ to indicate specific realizations of the random variables $$z, \, x$$. Probably a notation like $$x_i, z_i$$ is more common in general.

The notation/explanation is quite bad, because they should not say $$p_{\theta}(z)$$ is the distribution. $$p_{\theta}(z)$$, or if you wanted to be more precise, $$p_{\theta}(z^{(i)})$$, refers to the probability density.

• I noticed Kingma added notation guide in his other paper on VAEs, in appendix A. arxiv.org/pdf/1906.02691.pdf Seems like he uses the same notation for random vectors and their instances here too? Dec 2, 2021 at 18:06

You can read $$X=\{x^{(i)}\}_{i=1}^N$$ as $$X$$ represents the sequence of all values of $$x$$ from $$x_i$$ to $$x_N$$ where $$i$$ is all values from 1 to $$N$$.

To me, the notation is confusing since my experience tells me that curly braces are used for sets, but this seems to be the best interpretation.

• I guess its more like a sequence than a set? Dec 3, 2021 at 1:10
• Ah yes, that would be a good interpretation. Let me update the answer. Dec 3, 2021 at 11:31