5
$\begingroup$

Most of bibliography consider text classification as the classification of documents. When using bag of words and bayesian classification, they usually use the statistic TFIDF, where TF normalizes the word count with the number of words per document, and IDF focuses on ignoring widely used and thus useless words for this task.

My question is, why they keep the documents separated and create that statistic, if it is posible to merge all documents of the same class? This would have two advantages:

  • You can just use word counts instead of frequencies, as the documents per class label is 1.

  • Instead of using IDF, you just select features with enough standard deviation between classes.

$\endgroup$
0
$\begingroup$

Common words should not be ignored. For instance, there may be a correlation between document classes and whether more indefinite articles like a or an are included or more definite ones like the or this.

Neither is word count is a good substitution for word distribution. (Frequency though it is commonly (not necessarily frequently) used is an incorrect term. Words do not normally occur periodically in natural language.) Representing word distribution throughout the concatenated text of multiple documents of a single category.

The distribution of use of individual words and the distribution of language units and word pair proximity may correlate well to document classification, but the linguistic pattern distribution may vary from document to document within the same class, so the distribution of distributions is relevant. The deviations between distributions is information needed for proper statistical analysis.

In ten documents of differing sizes, the distribution of words may differ for some words and correlate for others. Concatenating them hides the distribution of distributions. From an information theory perspective, concatenating and totaling across the concatenation would represent an information loss with respect to separate documents analysis.

The naive Bayesian machinery produces the most reliable categorization when all information is integrated into the probability calculations.

$\endgroup$
0
$\begingroup$

My question is, why they keep the documents separated and create that statistic, if it is posible to merge all documents of the same class? This would have two advantages:

  • You can just use word counts instead of frequencies, as the documents per class label is 1.

In general, I don't think this is the case. I don't know if you have a specific equation in mind where it would end up being the same thing mathematically? Anyway, in general, it is possible that some documents in your corpus are very short, and others are very long. In such cases, you'd still want to make sure to use frequencies rather than raw word counts.

For example, suppose you have one very short text that is specifically about England. The word "England" may appear 10 times, but have a very high frequency due to it being a short text. If you compare it to a massive text that is about all countries in the world, that massive text may have the word "England" appearing 20 times, but with a significantly lower (relative) frequency.

  • Instead of using IDF, you just select features with enough standard deviation between classes.

I don't think this would work correctly because you may have significant differences among documents within a single class. Suppose, for example, that you have the following two classes of documents:

  1. Scientific articles (about AI, math, biology, linguistics, astronomy, whatever else you can think of...)
  2. News articles

Each of the "subdomains" in the single "scientific articles" class would likely have some highly specific terminology they use, which could be detected through TF-IDF. However, even though they're all in the same class of "scientific articles", they're likely all quite different from each other. If you put them all together and treat them as a single document, there is a risk that they'll all "average out" and become much more difficult to distinguish from a more general class such as the class of "news articles".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.