My question is, why they keep the documents separated and create that statistic, if it is posible to merge all documents of the same class? This would have two advantages:
- You can just use word counts instead of frequencies, as the documents per class label is 1.
In general, I don't think this is the case. I don't know if you have a specific equation in mind where it would end up being the same thing mathematically? Anyway, in general, it is possible that some documents in your corpus are very short, and others are very long. In such cases, you'd still want to make sure to use frequencies rather than raw word counts.
For example, suppose you have one very short text that is specifically about England. The word "England" may appear 10 times, but have a very high frequency due to it being a short text. If you compare it to a massive text that is about all countries in the world, that massive text may have the word "England" appearing 20 times, but with a significantly lower (relative) frequency.
- Instead of using IDF, you just select features with enough standard deviation between classes.
I don't think this would work correctly because you may have significant differences among documents within a single class. Suppose, for example, that you have the following two classes of documents:
- Scientific articles (about AI, math, biology, linguistics, astronomy, whatever else you can think of...)
- News articles
Each of the "subdomains" in the single "scientific articles" class would likely have some highly specific terminology they use, which could be detected through TF-IDF. However, even though they're all in the same class of "scientific articles", they're likely all quite different from each other. If you put them all together and treat them as a single document, there is a risk that they'll all "average out" and become much more difficult to distinguish from a more general class such as the class of "news articles".
