Why do we commonly use the $\log$ to squash frequencies?

Question

Term frequency and inverse document frequency are well-known terms in information retrieval.

I am presenting the definitions for both from p:12,13 of Vector Semantics and Embeddings

On term frequency

Term frequency is the frequency of the word $t$ in the term frequency document $d$. We can just use the raw count as the term frequency:

$$tf_{t, d} = \text{count}(t, d)$$

More commonly we squash the raw frequency a bit, by using the $\log_{10}$ of the frequency instead. The intuition is that a word appearing 100 times in a document doesn’t make that word 100 times more likely to be relevant to the meaning of the document.

On inverse document frequency

The $\text{idf}$ is defined using the fraction $\dfrac{N}{df_t}$, where $N$ is the total number of documents in the collection, and $\text{df}_t$ is the number of documents in which term $t$ occurs.......

Because of the large number of documents in many collections, this measure too is usually squashed with a log function. The resulting definition for inverse document frequency ($\text{idf}$) is thus

$$\text{idf}_t = \log_{10} \left(\dfrac{N}{df_t} \right)$$

If we observe the bolded portion of the quotes, it is evident that the $\log$ function is used commonly. It is not only used in these two definitions. It has been across many definitions in the literature. For example: entropy, mutual information, log-likelihood. So, I don't think squashing is the only purpose behind using the $\log$ function.

Is there any reason for selecting the logarithm function for squashing? Are there any advantages for $\log$ compared to any other squash functions, if available?

Answer 1

It's much easier to deal with logarithms, as the relevant numbers are usually very small or very large. If you have a long exponential expression, it's hard to see the difference, but if you're looking at 4.3 vs 5.6, you can immediately see what's happening. And logarithms are a well-known (and well-understood) way of achieving this compression. You can easily interpret the difference, depending on the base of the logarithm used.

Quite often the $log_2$ is used when you're dealing with entropy or information, as those are usually expressed in bits.

Answer 2

I would like to add details to Oliver's answer.

From the book "Pattern Recognition and Machine Learning" by Bishop (Section 1.2.5):

In practice, it is more convenient to maximize the log of the likelihood function. Because the logarithm is monotonically increasing function of its argument, maximization of the log of a function is equivalent to maximization of the function itself. Taking the log not only simplifies the subsequent mathematical analysis, but it also helps numerically because the product of a large number of small probabilities can easily underflow the numerical precision of the computer, and this is resolved by computing instead the sum of the log probabilities.

That is, $\log$ is monotonically increasing and hence preserves the order and the locations of the extrema. For instance, if $p(x) \geq p(y)$ then $\log\big(p(x)\big) \geq \log\big(p(y)\big)$ also holds. Therfore, maximizing likelihood is equivalent to maximizing log-likelihood.

Furthermore, it is extremely useful when calculating joint probabilities since a product can be replaced by a sum:

$$ \log \left(\prod_i P(x_i)\right) = \sum_i \log \left( P(x_i)\right) $$

This also makes calculation numerically stable and it is much easier to take a derivative of a sum of logarithms rather than to take a derivative of a product.