# How do very rare words tend to have very high PMI values?

Consider the following formulation for pointwise mutual information (PMI):

$$\text{PMI}(w, c) = \dfrac{p(w, c)}{p(w)p(c)}$$

Suppose there are $$W$$ words with $$C$$ context words. Then one can write in terms of frequency that

$$\text{PMI}(w, c) = \dfrac{\sum\limits_{i = 1}^{W} \sum\limits_{j = 1}^{C} f_{ij} }{\sum\limits_{i = 1}^{W}f_i \sum\limits_{j = 1}^{C} f_j}$$

I am going to calculate $$\text{PMI}(w, c)$$ for two different words and contexts based on the following table. The table is taken from fig 6.10 of this book. I calculated PMI for all pairs and tabulated below.

$$\begin{array}{|c|c|c|} \hline & \text{computer} & \text{data} & \text{result} & \text{pie} & \text{sugar} \\ \hline \text{cherry} & 8.2 \times 10^{-7} & 2.9 \times 10^{-6} & 3.9 \times 10^{-5} & 1.7 \times 10^{-3} & 8.4 \times 10^{-4} \\ \hline \text{strawberry} & 0 & 0 & 2.6 \times 10^{-5} & 1.4 \times 10^{-3} & 3.8 \times 10^{-3} \\ \hline \text{digital} & 9.6 \times 10^{-5} & 8.6 \times 10^{-5} & 5.2 \times 10^{-5} & 2.8 \times 10^{-6} & 1.9 \times 10^{-5} \\ \hline \text{information} & 8.6 \times 10^{-5} & 9.1 \times 10^{-5} & 1.03 \times 10^{-4} & 1.2 \times 10^{-6}& 2.7 \times 10^{-5}\\ \hline \end{array}$$

Based on the above values, we can also notice the following fact:

PMI has the problem of being biased toward infrequent events; very rare words tend to have very high PMI values.

However, it's unclear to me how this apparent behaviour is related to the mathematical formulation of the PMI above.

How do we understand the fact quoted above from the fractional form of PMI given by the equations above?

• "But, it is not intuitive for me to analyze the fact using the analytical form of PMI given on the top of this question." this seems to be worded quite confusingly. Jun 14, 2021 at 0:33
• @ThePointer I forget how to write equation numbers, so I told like that. Jun 14, 2021 at 1:00
• @hanugm I think that the confusing part in that sentence is that " analyze the fact using", because it's not super clear what "fact" you're referring to. Maybe you could rewrite that sentence as follows "However, it's unclear to me how this apparent behaviour is related to the mathematical formulation of the PMI above".
– nbro
Jun 14, 2021 at 1:34

Note: mutual information is typically expressed as a log value (usually $$log_2$$, as it's related to information), which makes them easier to compare -- you then don't have to worry about large exponential expressions with negative exponents.