Is logistic regression more free from the conditional independence assumption than naive Bayes?

Question

To my understanding, logistic regression is an extension of naive Bayes.

Suppose $X = \{x_1, x_2, \dots, x_N \}$ and $Y = \{0, 1\}$, each $x_i$ is i.i.d and $P(x_i \mid Y=y_k) \sim \mathcal{N}(\mu, \sigma^2)$ is a Gaussian distribution.

In order to create a linear decision surface, we take the assumption of each pdf $P(x_i \mid y_k)$ having variance $\sigma^2$ independent of the value of $Y$, i.e. $\sigma_{i, k} = \sigma_i$ ($i \rightarrow x_i, k \rightarrow y_k$).

Finally, we end up learning the coefficients $(\omega_0, \omega_i)$ that represent the linear decision surface in following equation (linear decision surface):

$$ \frac{P(Y=0 \mid X)}{P(Y=1 \mid X)} = \omega_0 + \sum_{i} \omega \cdot x_i $$

Even though the derivation of Linear Regression coefficients$(\omega_0, \omega_i)$ involves the assumption of conditional independent $x_i$ given $Y$, why is it said that learning these coefficients from training data are "somewhat more free" from conditional independence assumption, as compared to learning the regular Bayesian Distribution coefficients $\mu, \sigma$?

I came across this while following this course here.

Answer 1

I refer you to Prof. Tom Mitchell's (the lecturer in the video) draft chapter as the best explanation I could find. I will try to explain it in layman's terms here.

Given a boolean problem, our logistic regression classifier will assign the label $Y=0$ iff $w_0 + \sum_{i=1}^{n}w_{i}X_{i} > 0$ (from the video you posted; eqn. 18 in the draft chapter). Otherwise, it will assign the label $Y=1$. Remember the goal of logistic regression is to learn the weights $w_0, w_1, ..., w_n$ that will give us a linear function that would seperate our data best.

It can be shown that $w_0=\ln\frac{1-\pi}{\pi} + \sum_{i=1}^{n} \frac{\mu_{i1}^2 - \mu_{i0}^2}{2\sigma_{i}^2}$ and $w_i = \frac{\mu_{i0} - \mu_{i1}}{\sigma_{i}^2}$ (eqns. 19-23 in the draft chapter), where:

1. $\pi$ is $P(Y=1)$,
2. $\mu_{i0}$ and $\mu_{i1}$ are the means of feature $X_i$ in classes $Y=0$ and $Y=1$, respectively, and
3. $\sigma_i$ is the standard deviation of feature $X_i$

This means that we can express our weights $w_0, w_1, ..., w_n$ in terms of the Gaussian Naive Bayes classifier's parameters. Now, here is the difference:

Our goal is to find $P(Y=y_k|X_1 ... X_n)$. Assuming conditional independence, $P(Y=y_k|X_1 ... X_n) \propto P(Y=y_k) \prod_{i}^n P(X_i | Y=y_k)$.

If we estimate $\pi$, $\mu_{i0}$, $\mu_{i1}$, and $\sigma_i$, then use those values to estimate $w_0, w_1, ..., w_n$, then we use the conditional independence property directly in our calculations. This is what the Naive Bayes classifier does.

If we estimate $w_0, w_1, ..., w_n$ directly, by repeatedly minimizing the error of an assumed linear function, we are not strictly abiding by the conditional independence property, because we don't explicitly calculate an estimate of the conditional probability $P(X_i|Y=y_k)$. We completely bypass that step. This is what the Logistic Regression classifier does.

From the section 4 of the draft chapter (GNB is short for Gaussian Naive Bayes):

... Logistic Regression directly estimates the parameters of $P(Y|X)$, whereas Naive Bayes directly estimates parameters for $P(Y)$ and $P(X|Y)$... if the GNB assumptions hold, then asymptotically (as the number of training examples grows toward infinity) the GNB and Logistic Regression converge toward identical classifiers.

Another way to understand this is that it has to do with the fact that Logistic Regression is a discriminative classifier, whereas Naive Bayes is a generative classifier. For more on that, read Prof. Andrew Y. Ng and Prof. Michael I. Jordan's paper, On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes.