What is a probability distribution in machine learning?

Question

If we were learning or working in the machine learning field, then we frequently come across the term "probability distribution". I know what probability, conditional probability, and probability distribution/density in math mean, but what is its meaning in machine learning?

Take this example where $x$ is an element of $D$, which is a dataset,

$$x \in D$$

Let's say our dataset ($D$) is MNIST with about 70,000 images, so then $x$ becomes any image of those 70,000 images.

In many papers and web articles, these terms are often denoted as probability distributions

$$p(x)$$

or

$$p\left(z \mid x \right)$$

• What does $p(\cdot)$ even mean, and what kind of output does $p(\cdot)$ give?
• Is the output of $p(\cdot)$ a scalar, vector, or matrix?
• If the output is vector or matrix, then will the sum of all elements of this vector/matrix always be $1$?

This is my understanding,

$p(\cdot)$ is a function which maps the real distribution of the whole dataset $D$. Then $p(x)$ gives a scalar probability value given $x$, which is calculated from real distribution $p(\cdot)$. Similar to $p(H)=0.5$ in a coin toss experiment $D={\{H,T}\}$.

$p\left(z \mid x \right)$ is another function that maps the real distribution of the whole dataset to a vector $z$ given an input $x$ and the $z$ vector is a probability distribution that sums to $1$.

Are my assumptions correct?

An example would be a VAE's data generation process, which is represented in this equation

$$p_\theta(\mathbf{x}^{(i)}) = \int p_\theta(\mathbf{x}^{(i)}\vert\mathbf{z}) p_\theta(\mathbf{z}) d\mathbf{z}$$

Answer 1

Random variables

You do not necessarily need to understand the concept of a random variable (r.v.) to understand the concept of a probability distribution, but the concept of a random variable is strictly connected to the concept of a probability distribution (given that each random variable has an associated probability distribution), so, before proceeding, you should get familiar with the concept of an r.v., which is a (measurable) function from the sample space (the set of possible outcomes of an experiment) to a measurable space (you can ignore the definition of a measurable space and assume that the codomain of the random variable is a finite set of numbers).

Probability measure, cdf, pdf and pmf

The expression "probability distribution" can be ambiguous because it can be used to refer to different (even though related) mathematical concepts, such as probability measure, cumulative distribution function (c.d.f.), probability density function (p.d.f.), probability mass function (p.m.f.). If a person uses the expression "probability distribution", he (or she) intentionally (or not) refers to one or more of these mathematical concepts, depending on the context. However, a probability distribution is almost always a synonym for probability measure or c.d.f..

For example, if I say "Consider the Gaussian probability distribution", in that case, I could be referring to either the c.d.f. or the p.d.f. (or both) of the Gaussian distribution. Why couldn't I be referring to the p.m.f. of the Gaussian distribution? Because the Gaussian distribution is a continuous distribution, so it is a distribution associated with a continuous random variable, that is, a random variable that can take on continuous values (e.g. real numbers), so a Gaussian distribution does not have an associated p.m.f. or, in other words, no p.m.f. is defined for the Gaussian distribution. Why don't I simply say "Consider the p.d.f. of the Gaussian distribution." or "Consider a Gaussian p.d.f."? Because it is unnecessarily restrictive, given that, if I say "Consider the Gaussian distribution" I am implicitly also considering a p.d.f. and c.d.f. of the Gaussian distribution.

Similarly, in the case of a discrete distribution, such as the Bernoulli distribution, only the c.d.f. and p.m.f. are defined, so the Bernoulli distribution does not have an associated p.d.f.

However, it is important to recall that both continuous and discrete distributions have an associated c.d.f., so the expression "probability distribution" almost always (implicitly) refers to a c.d.f., which is defined based on a probability measure (as stated above).

Notation

In the same vein, the notation $p(x)$ can be as ambiguous as the expression "probability distribution", given that it can refer to different (but again related) concepts. However, $p(x)$ usually refers to a probability measure (so it refers to a probability distribution, given that a probability distribution is almost always a synonym for probability measure). In this case, assuming for simplicity that the r.v. is discrete, $p(x)$ is a shorthand for $p(X=x)$, which is also written as $\mathbb{P}(X=x)$ or $\operatorname{Pr}(X=x)$, where $X$ is a r.v., $x$ a realization of $X$ (that is, a value that the r.v. $X$ can take) and $X=x$ represents an event. Given that an r.v. is a function, the notation $X=x$ may look a bit weird.

In the case of a discrete r.v., $p(x)$ can also refer to a p.m.f. and it can be defined as $p_X(x) = \mathbb{P}(X=x)$ (I added the subscript $X$ to $p$ to emphasize that this is the p.m.f. of the discrete r.v. $X$). In the case of a continuous r.v., the p.d.f. is often denoted as $f$. In the case of both discrete and continuous r.v.s, the c.d.f is usually denoted with $F$ and it is defined as $F_X(x) = \mathbb{P}(X \leq x)$, where $\mathbb{P}$ is again a probability measure (or probability distribution). The p.d.f. of a continuous r.v. is then defined as the derivative of $F$. At this point, it should be clear why a probability distribution can refer to different but related concepts, but, in any case, it always refers to a probability measure.

Empirical distributions

There are also empirical distributions, which are distributions of the data that you have collected. For example, if you toss a coin 10 times, you will collect the results ("heads" or "tails"). You can count the number of times the coin landed on heads and tails, then you plot these numbers as a histogram, which essentially represents your empirical distribution, where the adjective "empirical" usually refers to the fact that there is an experiment involved.

Multivariate r.v.s and distributions

To complicate things even more, there are also multivariate random variables and probability distributions. However, all the concepts above more or less are also applicable in this case.

Parametrized distributions

A parametrized probability distribution, often denoted by $p_{\theta}$, is a family of probability distributions (defined by the parameters $\theta$), rather than a single probability distribution. For example, $\mathcal{N}(0, 1)$ refers to a single Gaussian distribution with zero mean and unit variance. However, $\mathcal{N}(\mu, \sigma)$, where $\theta=(\mu, \sigma)$ is a variable, is a family (or collection) of distributions.

Conclusion

To conclude, it is completely understandable that you are confused, given that the terminology and notation are used inconsistently, and there are several involved concepts, which I have not extensively covered in this answer (for example, I have not mentioned the concept of a probability space). If you get familiar with the concepts of probability measures, random variables, p.m.f., p.d.f., c.d.f., etc., and how they are related, then you will start to get a better feeling of the whole picture.

Answer 2

A probability distribution in ML is the same as a probability distribution elsewhere.

A probability distribution (or probability function, or probability mass function, or probability density function) is any function that accepts as input elements of some specific set $x \in X$, and produces as output, real-valued numbers between 0 and 1 (inclusive), such that $\int_{x \in X} p(x) = 1$ or for discrete sets, $\sum_{x \in X} p(x) = 1$.

These distributions can also be more complex. For example, a conditional probability distribution $P(Y|X)$ or a joint probability distribution $P(X,Y)$ accept more than one input, but again, are constrained to producing an output in the range 0 to 1, and to ensuring that the summation of the output over all possible inputs is exactly 1.

When these conditions are met, the functions output can be interpreted as a belief about the percentage of times the input event will occur, out of all events, or as a degree of believe in the input event having occurred vs. other events (i.e. it can be interpreted as a probability).