# Why do we commonly use the $\log$ to squash frequencies?

Term frequency and inverse document frequency are well-known terms in information retrieval.

I am presenting the definitions for both from p:12,13 of Vector Semantics and Embeddings

On term frequency

Term frequency is the frequency of the word $$t$$ in the term frequency document $$d$$. We can just use the raw count as the term frequency:

$$tf_{t, d} = \text{count}(t, d)$$

More commonly we squash the raw frequency a bit, by using the $$\log_{10}$$ of the frequency instead. The intuition is that a word appearing 100 times in a document doesn’t make that word 100 times more likely to be relevant to the meaning of the document.

On inverse document frequency

The $$\text{idf}$$ is defined using the fraction $$\dfrac{N}{df_t}$$, where $$N$$ is the total number of documents in the collection, and $$\text{df}_t$$ is the number of documents in which term $$t$$ occurs.......

Because of the large number of documents in many collections, this measure too is usually squashed with a log function. The resulting definition for inverse document frequency ($$\text{idf}$$) is thus

$$\text{idf}_t = \log_{10} \left(\dfrac{N}{df_t} \right)$$

If we observe the bolded portion of the quotes, it is evident that the $$\log$$ function is used commonly. It is not only used in these two definitions. It has been across many definitions in the literature. For example: entropy, mutual information, log-likelihood. So, I don't think squashing is the only purpose behind using the $$\log$$ function.

Is there any reason for selecting the logarithm function for squashing? Are there any advantages for $$\log$$ compared to any other squash functions, if available?

It's much easier to deal with logarithms, as the relevant numbers are usually very small or very large. If you have a long exponential expression, it's hard to see the difference, but if you're looking at 4.3 vs 5.6, you can immediately see what's happening. And logarithms are a well-known (and well-understood) way of achieving this compression. You can easily interpret the difference, depending on the base of the logarithm used.

Quite often the $$log_2$$ is used when you're dealing with entropy or information, as those are usually expressed in bits.

That is, $$\log$$ is monotonically increasing and hence preserves the order and the locations of the extrema. For instance, if $$p(x) \geq p(y)$$ then $$\log\big(p(x)\big) \geq \log\big(p(y)\big)$$ also holds. Therfore, maximizing likelihood is equivalent to maximizing log-likelihood.
$$\log \left(\prod_i P(x_i)\right) = \sum_i \log \left( P(x_i)\right)$$