I was reading a machine learning book that uses probabilities like these:

$P(x;y), P(x;y,z), P(x,y;z)$

I couldn't find what they are and how can I read and understand them?

Apart from the context, I saw one of these probabilities on it here:

  • $\begingroup$ These are conditional probabilities p(a;b) (probability of a when b occured) and probabilities for a and b together p(a,b) . Normally we mostly use Bayes rule in machine learning. So check out Bayes rule that will explain most of what you are looking for. $\endgroup$ Nov 6 '18 at 9:41
  • $\begingroup$ In the snippet you give (which looks like CS229), it simply denotes a parameterisation. p(y;eta) is the probability of y which depends on the (natural) parameter eta. (The above comment is not strictly correct, since eta might not have a well defined distribution in this setting.) $\endgroup$
    – user27182
    May 7 '19 at 10:57

This means "Parameterized by".

First, we all agree on the idea of conditional probabilities:

$$P(X | Y) = P(X,Y) / P(Y)$$

That is, the probability that X happens given that we've seen Y happen, is the fraction of worlds in which Y happened that also contain X. This is uncontroversial.

If you're a Bayesian, you might view parameters themselves as variables in a statistical model. So you might want to speak about the probability of a parameter taking on a certain value, or the probability of the data given that certain parameters have taken on certain values. In that case, you might write something like $P(D | \theta)$ to denote the probability of the data given a parameter $\theta$.

If you're a frequentist, you might find notation like this unsettling, because parameters don't have probabilities under the frequentist view, but instead have fixed values. You could talk about the probability of observing the data under a particular parameterization by defining a family of parameterized probability density functions, and then writing something like $P(D;\theta)$. You might also do this as a Bayesian if you wanted to clearly differentiate between model parameters and other things you might observe.

$P(x;y)$ then is read as "The probability of x under probability density function P, parameterized by y."

$P(x;y,z)$ is "The probability of x under probability density function P, parameterized by y and z."

$P(x,y;z)$ is "The probability of x and y jointly, under probability density function P, parameterized by z."

You could also write something like $P(X|Y; z)$, which would be "The conditional probability of X given that we observe Y, under probability density function P, parameterized by z."

An example of the latter would be in something like logistic regression. We might wish to know $P(L | D ; \theta)$, where L is the label of the data, D are the features we observe, and $\theta$ are the coefficients of the regression.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.