3
$\begingroup$

Term frequency and inverse document frequency are well-known terms in information retrieval.

I am presenting the definitions for both from p:12,13 of Vector Semantics and Embeddings

On term frequency

Term frequency is the frequency of the word $t$ in the term frequency document $d$. We can just use the raw count as the term frequency:

$$tf_{t, d} = \text{count}(t, d)$$

More commonly we squash the raw frequency a bit, by using the $\log_{10}$ of the frequency instead. The intuition is that a word appearing 100 times in a document doesn’t make that word 100 times more likely to be relevant to the meaning of the document.

On inverse document frequency

The $\text{idf}$ is defined using the fraction $\dfrac{N}{df_t}$, where $N$ is the total number of documents in the collection, and $\text{df}_t$ is the number of documents in which term $t$ occurs.......

Because of the large number of documents in many collections, this measure too is usually squashed with a log function. The resulting definition for inverse document frequency ($\text{idf}$) is thus

$$\text{idf}_t = \log_{10} \left(\dfrac{N}{df_t} \right)$$

If we observe the bolded portion of the quotes, it is evident that the $\log$ function is used commonly. It is not only used in these two definitions. It has been across many definitions in the literature. For example: entropy, mutual information, log-likelihood. So, I don't think squashing is the only purpose behind using the $\log$ function.

Is there any reason for selecting the logarithm function for squashing? Are there any advantages for $\log$ compared to any other squash functions, if available?

$\endgroup$
0
2
$\begingroup$

It's much easier to deal with logarithms, as the relevant numbers are usually very small or very large. If you have a long exponential expression, it's hard to see the difference, but if you're looking at 4.3 vs 5.6, you can immediately see what's happening. And logarithms are a well-known (and well-understood) way of achieving this compression. You can easily interpret the difference, depending on the base of the logarithm used.

Quite often the $log_2$ is used when you're dealing with entropy or information, as those are usually expressed in bits.

$\endgroup$
1
$\begingroup$

I would like to add details to Oliver's answer.

From the book "Pattern Recognition and Machine Learning" by Bishop (Section 1.2.5):

In practice, it is more convenient to maximize the log of the likelihood function. Because the logarithm is monotonically increasing function of its argument, maximization of the log of a function is equivalent to maximization of the function itself. Taking the log not only simplifies the subsequent mathematical analysis, but it also helps numerically because the product of a large number of small probabilities can easily underflow the numerical precision of the computer, and this is resolved by computing instead the sum of the log probabilities.

That is, $\log$ is monotonically increasing and hence preserves the order and the locations of the extrema. For instance, if $p(x) \geq p(y)$ then $\log\big(p(x)\big) \geq \log\big(p(y)\big)$ also holds. Therfore, maximizing likelihood is equivalent to maximizing log-likelihood.

Furthermore, it is extremely useful when calculating joint probabilities since a product can be replaced by a sum:

$$ \log \left(\prod_i P(x_i)\right) = \sum_i \log \left( P(x_i)\right) $$

This also makes calculation numerically stable and it is much easier to take a derivative of a sum of logarithms rather than to take a derivative of a product.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.