# How many parameter would there be in a logistic regression model used to classify reviews into "good" or "bad"?

Suppose we want to classify a review as good ($$1$$) or bad ($$0$$). We have a training data set of $$10,000$$ reviews. Also, suppose we have a vocabulary of $$100,000$$ words $$w_1, \dots, w_{100,000}$$. So the data is a matrix of dimension $$100,000 \times 10,000$$. Let's represent each of the words in the reviews using a bag-of-words approach over tf-idf values. Also, we normalize the rows such that they sum to $$1$$.

In a logistic regression approach, would we have $$10,000$$ different logistic regression models as follows:

$$\log \left(\frac{p}{1-p} \right)_{1} = \beta_{0_{1}} + \beta_{1_{1}}w_{11} + \dots + \beta_{100,000_{1}}w_{100,000} \\ \vdots \\ \log \left(\frac{p}{1-p} \right)_{10,000} = \beta_{0_{10,000}} + \beta_{1_{10,000}}w_{11} + \dots + \beta_{100,000_{10,000}}w_{100,000}$$

So are we estimating $$100,000 \times 10,000$$ coefficients?

## 1 Answer

Nope! Our number of coefficients will be driven by the vocabulary, and we'll use each of those 10K samples to estimate values for those coefficients - so, 'just' 100K samples. However, word frequency in human languages follows a Zipf distribution => most of those words will be rare, seen in only a few samples (=> won't even be able to determine whether they are useful or not, let alone get good values for coefficients). For an application like this one, you would probably find most of the value from a few hundred words.