# How do very rare words tend to have very high PMI values?

Consider the following formulation for pointwise mutual information (PMI):

$$\text{PMI}(w, c) = \dfrac{p(w, c)}{p(w)p(c)}$$

Suppose there are $$W$$ words with $$C$$ context words. Then one can write in terms of frequency that

$$\text{PMI}(w, c) = \dfrac{\sum\limits_{i = 1}^{W} \sum\limits_{j = 1}^{C} f_{ij} }{\sum\limits_{i = 1}^{W}f_i \sum\limits_{j = 1}^{C} f_j}$$

I am going to calculate $$\text{PMI}(w, c)$$ for two different words and contexts based on the following table. The table is taken from fig 6.10 of this book.

I calculated PMI for all pairs and tabulated below.

$$\begin{array}{|c|c|c|} \hline & \text{computer} & \text{data} & \text{result} & \text{pie} & \text{sugar} \\ \hline \text{cherry} & 8.2 \times 10^{-7} & 2.9 \times 10^{-6} & 3.9 \times 10^{-5} & 1.7 \times 10^{-3} & 8.4 \times 10^{-4} \\ \hline \text{strawberry} & 0 & 0 & 2.6 \times 10^{-5} & 1.4 \times 10^{-3} & 3.8 \times 10^{-3} \\ \hline \text{digital} & 9.6 \times 10^{-5} & 8.6 \times 10^{-5} & 5.2 \times 10^{-5} & 2.8 \times 10^{-6} & 1.9 \times 10^{-5} \\ \hline \text{information} & 8.6 \times 10^{-5} & 9.1 \times 10^{-5} & 1.03 \times 10^{-4} & 1.2 \times 10^{-6}& 2.7 \times 10^{-5}\\ \hline \end{array}$$

Based on the above values, we can also notice the following fact:

PMI has the problem of being biased toward infrequent events; very rare words tend to have very high PMI values.

However, it's unclear to me how this apparent behaviour is related to the mathematical formulation of the PMI above.

How do we understand the fact quoted above from the fractional form of PMI given by the equations above?

• "But, it is not intuitive for me to analyze the fact using the analytical form of PMI given on the top of this question." this seems to be worded quite confusingly. Jun 14, 2021 at 0:33
• @ThePointer I forget how to write equation numbers, so I told like that. Jun 14, 2021 at 1:00
• @hanugm I think that the confusing part in that sentence is that " analyze the fact using", because it's not super clear what "fact" you're referring to. Maybe you could rewrite that sentence as follows "However, it's unclear to me how this apparent behaviour is related to the mathematical formulation of the PMI above".
– nbro
Jun 14, 2021 at 1:34

## 1 Answer

Note: mutual information is typically expressed as a log value (usually $$log_2$$, as it's related to information), which makes them easier to compare -- you then don't have to worry about large exponential expressions with negative exponents.

The reason for the bias is in the distributional properties of words. A rare word will have a small frequency of occurrence (by definition), so multiplied with a more common word, the denominator will be fairly small. But the numerator will also be very small, as there aren't that many opportunities for the rare word to occur near the more common word.

However, while with two common words the nominator will be much higher (more co-occurrences), the denominator will now — compared to the rare/common instance — be several orders of magnitude larger. Thus the overall value will be smaller.

Rare co-occurrences are overly dependent on chance, as the nominator can easily fluctuate randomly (say, 1 or 2 co-occurences), and with a comparatively smaller denominator that can make a big difference.

In linguistics, you would generally ignore mutual information values above a certain threshold, as the values are just too unreliable to be meaningful. In fact, when I was still working in academia, mutual information was increasingly replaced by other metrics, such as log-likelihood, which were more robust.

• Although it is okay to understand. If possible, please keep in brackets which one you are referring as word and which one as context. Jun 14, 2021 at 23:32