2
$\begingroup$

Usually, I see the conventions:

  • discrete random variable is denoted as $X$,
  • the pmf is written as $P(X=x)$ or $p(X=x)$ or $p_{X}(x)$ or $p(x)$, where $x$ is an instance of $X$
  • a continuous random variable is denoted as $X$,
  • the pdf is denoted as $f_{X}(x)$ or $f(x)$, where $x$ is an instance of $X$; sometimes $p$ is used here too instead of $f$.

However, the VAE paper uses slightly different notation that I'm trying to understand

Let us consider some dataset $\mathbf{X}=\left\{\mathbf{x}^{(i)}\right\}_{i=1}^{N}$ consisting of $N$ i.i.d. samples of some continuous or discrete variable $\mathrm{x}$. We assume that the data are generated by some random process, involving an unobserved continuous random variable $\mathbf{z}$. The process consists of two steps: (1) a value $\mathbf{z}^{(i)}$ is generated from some prior distribution $p_{\boldsymbol{\theta}^{*}}(\mathbf{z}) ;(2)$ a value $\mathbf{x}^{(i)}$ is generated from some conditional distribution $p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$. We assume that the prior $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ and likelihood $p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$ come from parametric families of distributions $p_{\boldsymbol{\theta}}(\mathbf{z})$ and $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$, and that their PDFs are differentiable almost everywhere w.r.t. both $\boldsymbol{\theta}$ and $\mathbf{z}$. Unfortunately, a lot of this process is hidden from our view: the true parameters $\theta^{*}$ as well as the values of the latent variables $\mathrm{z}^{(i)}$ are unknown to us.

So I am looking at these:

  • $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$
  • $p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$
  • dataset $\mathbf{X}=\left\{\mathbf{x}^{(i)}\right\}_{i=1}^{N}$

So I know the subscript for $\theta$ denotes those are the parameters for the pdf. It says "discrete variable $\mathrm{x}$", "unobserved continuous random variable $\mathbf{z}$", and "latent variables $\mathrm{z}^{(i)}$". In the top, where I wrote " discrete random variable $X$", seems like that's the equivalent of "discrete variable $\mathrm{x}$" in this paper.

So, it looks like they're writing the PDFs as a function of the random variables. Is my assumption correct? Because it is different than the typical conventions I see.

edit: looks like his other paper has a notation guide, in the appendix, though it seems like he's conflating both random vector and instances of vector in the notation? https://arxiv.org/pdf/1906.02691.pdf

$\endgroup$
4
  • 1
    $\begingroup$ no they’re not writing the pdf/pmf as a function of a random variable. The capital X here is just denoting your dataset of realisations of random variables $\endgroup$ Nov 30, 2021 at 23:04
  • $\begingroup$ Sorry I rewrote my comment, that particular part about p(X) might have made things more confusing. $\endgroup$
    – a12345
    Nov 30, 2021 at 23:17
  • 1
    $\begingroup$ I would say that here they’re saying ‘discrete variable’ to denote a realisation of the random variable, so $p(x)$ in this setting is still in line with standard notation. $p(\cdot)$ is a function of $x$ (discrete in this case), it’s just that $x$ are some random realisations of a random variable $X$. $\endgroup$ Nov 30, 2021 at 23:28
  • $\begingroup$ One other possibility I'm considering is that they could just be using it for both the instance and random variable? $\endgroup$
    – a12345
    Dec 1, 2021 at 0:39

3 Answers 3

4
$\begingroup$

When it comes to notation/terminology, often, people in machine learning are (a bit?) sloppy, which causes a lot of confusion, especially for newcomers to the field or people not very math-savvy. I was also confused about this notation at some point (see my last questions here, which are all about this confusing topic). See also this answer.

In the VAE paper, $\mathbf{X}$ is a dataset, as the authors write.

Your confusion also arises because the authors vaguely use the term "probability distribution", rather than pdf or pmf, to refer, for example, to $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$, which thus does not refer to a pdf or pmf. In fact, the authors also write

their PDFs are differentiable almost everywhere w.r.t. both $\boldsymbol{\theta}$ and $\mathbf{z}$

The $\mathbf{z}$ can refer to

  1. a random variable, or
  2. an input to the function $p_{\boldsymbol{\theta}^{*}}$

If it's the first case, then $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ is the composition of 2 functions (because a rv is also a function).

If it's the second case, then $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ is the evaluation of $p_{\boldsymbol{\theta}^{*}}$ at $\mathbf{z}$.

I think the 2nd case is the most likely. In addition, people are being sloppy here and use the notation $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ (rather than just $p_{\boldsymbol{\theta}^{*}}$) to emphasize $p_{\boldsymbol{\theta}^{*}}$ is a function of some input variable (not random variable!), which we denote with the letter $\mathbf{z}$ to remind ourselves that $\mathbf{z}$ is associated with a random variable denoted with the same letter (and maybe also in bold and lowercase).

So, in this case, let's say we denote the random variable associated with $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ with $\mathbf{z}$, then we could refer to this associated prior more explicitly as follows $p_\mathbf{z}(\mathbf{z})$ (but that would even be more confusing). It would have been a better idea to use $\mathbf{Z}$, but then we may use the upper case letters to denote matrices or sets (like the VAE paper), so we end up with this mess (which is one of the 2 mythical difficult problems well-known in Computer Science, i.e. naming things), which we need to learn to deal with or just ignore.

Conclusion: when I look at $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$, which has been referred to as a probability distribution, I think there's also some associated random variable, which people, in that same context, will probably denote as $\mathbf{z}$ or $\mathbf{Z}$. There may also be some input variable (not a random variable), which we denote by $\mathbf{z}$ or $z$. If they are not mentioned, then I just ignore that. I never think that $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ is the composition of 2 functions (even if that's the case), because that case was never useful in my readings.

$\endgroup$
2
  • $\begingroup$ I just noticed in Appendix A of his other paper on VAEs, he goes over the notation. Seems like he's using the same notation for random vectors and instances of random vectors. arxiv.org/pdf/1906.02691.pdf $\endgroup$
    – a12345
    Dec 2, 2021 at 17:58
  • $\begingroup$ and yeah, figuring out which notation to use is a mess. I was thinking italics, using mathcal in latex , and something like this:tex.stackexchange.com/questions/520412/… gives more variety there. $\endgroup$
    – a12345
    Dec 2, 2021 at 18:03
2
$\begingroup$

Machine learning papers are often somewhat confused about the distinction between a distribution and its probability density. I would rewrite this

The process consists of two steps: (1) a value $\mathbf{z}^{(i)}$ is generated from some prior distribution $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$; (2) a value $\mathbf{x}^{(i)}$ is generated from some conditional distribution $p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$.

as follows

The process consists of two steps: (1) a value $\mathbf{z}^{(i)}$ is generated from some prior distribution. The probability density of selecting $\mathbf{z}^{(i)}$ is known and denoted as $p_{\boldsymbol{\theta}^{*}}(\mathbf{z}^{(i)})$. (2) a value $\mathbf{x}^{(i)}$ is generated from some conditional distribution. The probability density of selecting $\mathbf{x}^{(i)}$ given $\mathbf{z}^{(i)}$ is known and denoted as $p_{\boldsymbol{\theta}^{*}}(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i)})$.

As for the uppercase/lower case notation, this notation is not used in machine learning. $z, \, x$ are both random variables. In this paper, the authors use $z^{(i)}$, $x^{(i)}$ to indicate specific realizations of the random variables $z, \, x$. Probably a notation like $x_i, z_i$ is more common in general.

The notation/explanation is quite bad, because they should not say $p_{\theta}(z)$ is the distribution. $p_{\theta}(z)$, or if you wanted to be more precise, $p_{\theta}(z^{(i)})$, refers to the probability density.

$\endgroup$
1
  • $\begingroup$ I noticed Kingma added notation guide in his other paper on VAEs, in appendix A. arxiv.org/pdf/1906.02691.pdf Seems like he uses the same notation for random vectors and their instances here too? $\endgroup$
    – a12345
    Dec 2, 2021 at 18:06
1
$\begingroup$

You can read $X=\{x^{(i)}\}_{i=1}^N$ as $X$ represents the sequence of all values of $x$ from $x_i$ to $x_N$ where $i$ is all values from 1 to $N$.

To me, the notation is confusing since my experience tells me that curly braces are used for sets, but this seems to be the best interpretation.

$\endgroup$
2
  • $\begingroup$ I guess its more like a sequence than a set? $\endgroup$
    – a12345
    Dec 3, 2021 at 1:10
  • $\begingroup$ Ah yes, that would be a good interpretation. Let me update the answer. $\endgroup$ Dec 3, 2021 at 11:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .