1
$\begingroup$

While studying about the n-gram models, I encountered the terms "statistical model" and "probabilistic model" several times.

I got a basic doubt that will there be any probabilistic model that is not statistical restricted to models that works on datasets.

In machine learning, we use datasets. Any model that uses dataset can be called as a statistical model since statistics is a branch of mathematics that tries to find insights related to data.

All the models that calculates probabilities using datasets, for any task, are called (empirical) probabilistic models.

Thus, if I am not wrong, every probabilistic model has to be a statistical model since it uses data. Am I wrong?

Is there any model in literature that is a statistical model but not probabilistic?

$\endgroup$
3
$\begingroup$

First of all, I don't know of any textbook that clarifies these terms, but, although I am not a statistician, in addition to the other answer, one possible way to look at it is as follows.

You use probability theory to model your problem. For example, if it's a classification problem, you could define the conditional probability distribution $p(y \mid x)$, which would compute the probability of a label $y$ given an input $x$. In other words, you assume that there is a probability distribution of the form $p(y \mid x)$ that generates your data, so here $p$ is the "model". If you want to generate images, for instance, you could model the process that generates them as the marginal distribution $p(x)$, which, ideally, would tell you the probability of sampling a specific image $x$. This is the theory. So, no observed data is still involved here. By the way, this is exactly how people usually model the machine learning problems in the sub-field of statistical learning theory.

In practice, you need to estimate these probability distributions. To estimate them, you can use data. The type of data depends, of course, on the problem and model. For example, in the case of $p(y \mid x)$, you may need a labelled dataset $D$. So, if you estimate $p(y \mid x)$ with $D$ to obtain $\hat{p}(y \mid x)$, then $\hat{p}$ would be a statistical model, in the sense that you estimated it from observed/empirical data. In general, statistics is all about taking data and using it to build "models" that can be used for prediction or forecasting (of future inputs) or inference (i.e. understanding the properties of the data-generating process or probability distribution) or just to compute the so-called "statistics" (hence the name of the field!), such as the "sample average" (i.e. the average of your observe data points, where the "sample" here refers to your dataset of points, which are also sometimes known as "samples", just to make things even more confusing!)

So, let me address your questions and comments, but take my comments below with a grain of salt, because I am not a statistician.

All the models that calculates probabilities using datasets, for any task, are called (empirical) probabilistic models.

To me, this would be a reasonable statement. In this example, you seem to be talking about $\hat{p}(y \mid x)$, which I would also call a probabilistic model, although it's just an estimate of the theoretical/ideal one.

Is there any model in literature that is a statistical model but not probabilistic?

If we follow my reasoning above, initially, if you do not explicitly model your problem as the estimation of some probability distribution that generated the data, then we would be estimating something from data (so we would be building a statistical model), but it wouldn't be clear whether this "statistical model" is an estimate of some theoretical/probabilistic one. So, I don't really have a definitive answer to your question. I suppose that any statistical model could be modelled with the tools of probability theory, so I would be more inclined to think that the answer to your question is "no".

In addition to what I just said above, if you take a book like Machine Learning: A Probabilistic Perspective, here are a few examples of how the author uses the terms "statistical model" and "probabilistic model". For example, he writes (section 7.3, page 217)

A common way to estimate the parameters of a statistical model is to compute the MLE, which is defined as

$$ \hat{\boldsymbol{\theta}} \triangleq \arg \max _{\boldsymbol{\theta}} \log p(\mathcal{D} \mid \boldsymbol{\theta}) $$

So, in this case, is $p$ a statistical or probabilistic model, according to my definitions above? Of course, ignoring the potentially different notation being used here to refer to a statistical model, i.e. without the $\hat{}$, I think that this $p$ could be considered a statistical model (in the sense that $\hat{\boldsymbol{\theta}}$ would be estimated from the observed dataset $\mathcal{D}$, assuming it's the observed dataset and not some random variable) but at the same time also a probabilistic one, in the sense that, here, we are assuming that we have some kind of "theoretical likelihood". In any case, the likelihood is something that can make this discussion even more confusing, because the likelihood is not really a probability distribution (if you integrate with respect to the parameters). In any case, here, you could consider $p(\mathcal{D} \mid \boldsymbol{\theta})$ as a (Bayesian) probabilistic model, i.e. you assume that there's some parameters that generate the data and, if you consider it as a conditional probability distribution over the data, rather than the parameters, then this would be consistent with what I said above.

Here's another example (section 1.3.1, p. 10).

In this book, we focus on model based clustering, which means we fit a probabilistic model to the data, rather than running some ad hoc algorithm.

This usage also seems consistent with my description above. Here, I interpret the part "we fit a probabilistic model to the data" as "we estimate the probability distribution given the data".

Or, in section 1.4.1 (p. 16)

In this book, we will be focussing on probabilistic models of the form $p(y \mid x)$ or $p(x)$, depending on whether we are interested in supervised or unsupervised learning respectively.

The discussion can become even more complicated, if we start to consider parametric vs non-parametric models, which are mentioned in that same section, where you make or not assumptions about the data-generating process.

So, to conclude, I think that these terms are often used vaguely and sometimes interchangeably, so the confusion is normal.

$\endgroup$
1
$\begingroup$

It is purely terminological. A probabilistic model uses probabilities, but one usually does not know what the 'correct' probabilities are, or where they come from. This is where statistics comes into play:

You can estimate probabilities from (empirical) data[*]. For example, if you develop a probabilisitc parts-of-speech tagger, you need probabilities typically for the probability of a certain word to be of a particular class, and for transition probabilities, ie how likely is it for tag a to be followed by tag b rather than tag c. So you might devise an equation that states that the probability of a token being assigned a particular tag is the product of the probability of the tag given the word and the tag given the previous two tags.

But you don't know what these probabilities are, and you cannot derive them from any formula. Instead, you get these values by looking at your training data, ie you count how often each event occurs, and normalise it to be in the range $[0..1]$.

In practice therefore, probabilities and statistical likelihoods are pretty much identical; probabilities are the theoretical values used in your model, and the actual values are derived using statistics. To make clear in equations that they aren't strictly the same, probabilities are usually denoted by $p$, whereas estimates based on statistics are marked $\hat{p}$.

[*] Another way to get probabilities is to calculate them, but looking at large-scale open-ended problems this is not easily possible. That's why you take samples to estimate the values -- this is then called training data.

$\endgroup$
4
  • $\begingroup$ You can compute statistics from the data, such as the sample mean, and, although I am not a statistician, it doesn't sound very natural to me to say "statistical data", but it makes some sense to say "empirical data", although it sounds redundant, as data is usually empirical, in the sense that it's something that we collect/observe and it's not just some theoretical thing. $\endgroup$
    – nbro
    Jul 1 at 12:41
  • $\begingroup$ I would also remove the sentence "To a non-mathematician therefore probabilities and statistical likelihoods are pretty much identical". To me it doesn't make much sense to talk about "probabilistic likelihood" or "statistical likelihood". You define a probabilistic model, e.g. $p(y \mid x)$ or the likelihood, then you try to maximize it. So, you model your problem with probability theory, but then you estimate a statistical model, because the "model" is based on the data. $\endgroup$
    – nbro
    Jul 1 at 12:43
  • $\begingroup$ In any case, this is just one interpretation and, as you say, it's definitely not clear from the literature and books if there's any consensus on these definitions. Apart from these 2 things, I like your answer. I think it $\endgroup$
    – nbro
    Jul 1 at 12:43
  • $\begingroup$ @nbro Thanks for your comments, I have amended my answer. $\endgroup$ Jul 1 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.