2
$\begingroup$

Usually, I see the conventions:

  • discrete random variable is denoted as $X$,
  • the pmf is written as $P(X=x)$ or $p(X=x)$ or $p_{X}(x)$ or $p(x)$, where $x$ is an instance of $X$
  • a continuous random variable is denoted as $X$,
  • the pdf is denoted as $f_{X}(x)$ or $f(x)$, where $x$ is an instance of $X$; sometimes $p$ is used here too instead of $f$.

However, the VAE paper uses slightly different notation that I'm trying to understand

Let us consider some dataset $\mathbf{X}=\left\{\mathbf{x}^{(i)}\right\}_{i=1}^{N}$ consisting of $N$ i.i.d. samples of some continuous or discrete variable $\mathrm{x}$. We assume that the data are generated by some random process, involving an unobserved continuous random variable $\mathbf{z}$. The process consists of two steps: (1) a value $\mathbf{z}^{(i)}$ is generated from some prior distribution $p_{\boldsymbol{\theta}^{*}}(\mathbf{z}) ;(2)$ a value $\mathbf{x}^{(i)}$ is generated from some conditional distribution $p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$. We assume that the prior $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ and likelihood $p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$ come from parametric families of distributions $p_{\boldsymbol{\theta}}(\mathbf{z})$ and $p_{\boldsymbol{\theta}}(\mathbf{x} \mid \mathbf{z})$, and that their PDFs are differentiable almost everywhere w.r.t. both $\boldsymbol{\theta}$ and $\mathbf{z}$. Unfortunately, a lot of this process is hidden from our view: the true parameters $\theta^{*}$ as well as the values of the latent variables $\mathrm{z}^{(i)}$ are unknown to us.

So I am looking at these:

  • $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$
  • $p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$
  • dataset $\mathbf{X}=\left\{\mathbf{x}^{(i)}\right\}_{i=1}^{N}$

So I know the subscript for $\theta$ denotes those are the parameters for the pdf. It says "discrete variable $\mathrm{x}$", "unobserved continuous random variable $\mathbf{z}$", and "latent variables $\mathrm{z}^{(i)}$". In the top, where I wrote " discrete random variable $X$", seems like that's the equivalent of "discrete variable $\mathrm{x}$" in this paper.

So, it looks like they're writing the PDFs as a function of the random variables. Is my assumption correct? Because it is different than the typical conventions I see.

edit: looks like his other paper has a notation guide, in the appendix, though it seems like he's conflating both random vector and instances of vector in the notation? https://arxiv.org/pdf/1906.02691.pdf

$\endgroup$
4
  • 1
    $\begingroup$ no they’re not writing the pdf/pmf as a function of a random variable. The capital X here is just denoting your dataset of realisations of random variables $\endgroup$ Nov 30 '21 at 23:04
  • $\begingroup$ Sorry I rewrote my comment, that particular part about p(X) might have made things more confusing. $\endgroup$
    – a12345
    Nov 30 '21 at 23:17
  • 1
    $\begingroup$ I would say that here they’re saying ‘discrete variable’ to denote a realisation of the random variable, so $p(x)$ in this setting is still in line with standard notation. $p(\cdot)$ is a function of $x$ (discrete in this case), it’s just that $x$ are some random realisations of a random variable $X$. $\endgroup$ Nov 30 '21 at 23:28
  • $\begingroup$ One other possibility I'm considering is that they could just be using it for both the instance and random variable? $\endgroup$
    – a12345
    Dec 1 '21 at 0:39
4
$\begingroup$

When it comes to notation/terminology, often, people in machine learning are (a bit?) sloppy, which causes a lot of confusion, especially for newcomers to the field or people not very math-savvy. I was also confused about this notation at some point (see my last questions here, which are all about this confusing topic). See also this answer.

In the VAE paper, $\mathbf{X}$ is a dataset, as the authors write.

Your confusion also arises because the authors vaguely use the term "probability distribution", rather than pdf or pmf, to refer, for example, to $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$, which thus does not refer to a pdf or pmf. In fact, the authors also write

their PDFs are differentiable almost everywhere w.r.t. both $\boldsymbol{\theta}$ and $\mathbf{z}$

The $\mathbf{z}$ can refer to

  1. a random variable, or
  2. an input to the function $p_{\boldsymbol{\theta}^{*}}$

If it's the first case, then $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ is the composition of 2 functions (because a rv is also a function).

If it's the second case, then $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ is the evaluation of $p_{\boldsymbol{\theta}^{*}}$ at $\mathbf{z}$.

I think the 2nd case is the most likely. In addition, people are being sloppy here and use the notation $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ (rather than just $p_{\boldsymbol{\theta}^{*}}$) to emphasize $p_{\boldsymbol{\theta}^{*}}$ is a function of some input variable (not random variable!), which we denote with the letter $\mathbf{z}$ to remind ourselves that $\mathbf{z}$ is associated with a random variable denoted with the same letter (and maybe also in bold and lowercase).

So, in this case, let's say we denote the random variable associated with $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ with $\mathbf{z}$, then we could refer to this associated prior more explicitly as follows $p_\mathbf{z}(\mathbf{z})$ (but that would even be more confusing). It would have been a better idea to use $\mathbf{Z}$, but then we may use the upper case letters to denote matrices or sets (like the VAE paper), so we end up with this mess (which is one of the 2 mythical difficult problems well-known in Computer Science, i.e. naming things), which we need to learn to deal with or just ignore.

Conclusion: when I look at $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$, which has been referred to as a probability distribution, I think there's also some associated random variable, which people, in that same context, will probably denote as $\mathbf{z}$ or $\mathbf{Z}$. There may also be some input variable (not a random variable), which we denote by $\mathbf{z}$ or $z$. If they are not mentioned, then I just ignore that. I never think that $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$ is the composition of 2 functions (even if that's the case), because that case was never useful in my readings.

$\endgroup$
2
  • $\begingroup$ I just noticed in Appendix A of his other paper on VAEs, he goes over the notation. Seems like he's using the same notation for random vectors and instances of random vectors. arxiv.org/pdf/1906.02691.pdf $\endgroup$
    – a12345
    Dec 2 '21 at 17:58
  • $\begingroup$ and yeah, figuring out which notation to use is a mess. I was thinking italics, using mathcal in latex , and something like this:tex.stackexchange.com/questions/520412/… gives more variety there. $\endgroup$
    – a12345
    Dec 2 '21 at 18:03
2
$\begingroup$

Machine learning papers are often somewhat confused about the distinction between a distribution and its probability density. I would rewrite this

The process consists of two steps: (1) a value $\mathbf{z}^{(i)}$ is generated from some prior distribution $p_{\boldsymbol{\theta}^{*}}(\mathbf{z})$; (2) a value $\mathbf{x}^{(i)}$ is generated from some conditional distribution $p_{\boldsymbol{\theta}^{*}}(\mathbf{x} \mid \mathbf{z})$.

as follows

The process consists of two steps: (1) a value $\mathbf{z}^{(i)}$ is generated from some prior distribution. The probability density of selecting $\mathbf{z}^{(i)}$ is known and denoted as $p_{\boldsymbol{\theta}^{*}}(\mathbf{z}^{(i)})$. (2) a value $\mathbf{x}^{(i)}$ is generated from some conditional distribution. The probability density of selecting $\mathbf{x}^{(i)}$ given $\mathbf{z}^{(i)}$ is known and denoted as $p_{\boldsymbol{\theta}^{*}}(\mathbf{x}^{(i)} \mid \mathbf{z}^{(i)})$.

As for the uppercase/lower case notation, this notation is not used in machine learning. $z, \, x$ are both random variables. In this paper, the authors use $z^{(i)}$, $x^{(i)}$ to indicate specific realizations of the random variables $z, \, x$. Probably a notation like $x_i, z_i$ is more common in general.

The notation/explanation is quite bad, because they should not say $p_{\theta}(z)$ is the distribution. $p_{\theta}(z)$, or if you wanted to be more precise, $p_{\theta}(z^{(i)})$, refers to the probability density.

$\endgroup$
1
  • $\begingroup$ I noticed Kingma added notation guide in his other paper on VAEs, in appendix A. arxiv.org/pdf/1906.02691.pdf Seems like he uses the same notation for random vectors and their instances here too? $\endgroup$
    – a12345
    Dec 2 '21 at 18:06
1
$\begingroup$

You can read $X=\{x^{(i)}\}_{i=1}^N$ as $X$ represents the sequence of all values of $x$ from $x_i$ to $x_N$ where $i$ is all values from 1 to $N$.

To me, the notation is confusing since my experience tells me that curly braces are used for sets, but this seems to be the best interpretation.

$\endgroup$
2
  • $\begingroup$ I guess its more like a sequence than a set? $\endgroup$
    – a12345
    Dec 3 '21 at 1:10
  • $\begingroup$ Ah yes, that would be a good interpretation. Let me update the answer. $\endgroup$ Dec 3 '21 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.