3
$\begingroup$

I'm reading about how Conditional Probability/ Bayes Theorem is used in Naive Bayes in Intro to Statistical Learning, but it seems like it isn't that "groundbreaking" as it is described?

If I'm not mistaken doesn't every single ML classifier use conditional probability/Bayes in its underlying assumptions, not just Naive Bayes? We are always trying to find the most likely class/label, given a set of features. And we can only deduce that using Bayes rule since we are (usually) solving for P(class|features) with P(features|class)?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

Conditional probability and Bayes rule are related but they are not the same thing, you can predict conditional probabilities without using Bayes rule.

So no, not all machine learning classifiers use Bayes rule, standard neural networks do not use Bayes rule at all, SVMs and linear classifiers neither.

A better counterexample is Bayesian Neural Networks, which have a probability distribution over the weights, and Bayes rule is used during learning and inference, these are not the same as standard neural networks.

As reference for this statement, I leave the following quote from Section 3.1 of the the paper Uncertainty Quantification for Deep Neural Networks: An Empirical Comparison and Usage Guidelines:

BNNs are neural networks with probabilistic weights, instead of scalar weights as in PPNN, and are represented as probability density functions. To train a BNN, first, a prior distribution p(θ) over weights θ has to be defined. Then, given some data D, the posterior distribution p(θ|D), i.e., the trained BNN is inferred using Bayes rule:

Improve this answer
$\endgroup$
19
  • $\begingroup$ In practice, BNNs (that I most familiar with) do not directly use Bayes rules to learn the posterior. They use variational inference (i.e. formulate the inference problem as an optimization problem). So, it's not correct to say that the Bayes rule is used during inference and training of (all) BNNs. $\endgroup$
    – nbro
    Dec 24, 2022 at 10:41
  • $\begingroup$ @nbro What you are saying is basically, there is one kind of BNN that uses VI, which means all BNNs use VI, which is incorrect. Even with VI you use the predictive posterior distribution to make prediction which implicitly uses Bayes rule. $\endgroup$
    – Dr. Snoopy
    Dec 24, 2022 at 13:53
  • $\begingroup$ I'm not saying there's one kind of BNNs and that all BNNs use VI. I said "BNNs (that I most familiar with)". Maybe the parentheses were not necessary. In other words, I am saying that some BNNs do not use Bayes rule. Your last paragraph seems to suggest that all BNNs use the Bayes rule, which is incorrect. With VI, you do not really use any Bayes rule. You're solving an optimization problem, which is equivalent (up to a constant) to solving the inference problem using the Bayes rule by finding the integrals (with Monte Carlo methods or in closed-form) $\endgroup$
    – nbro
    Dec 24, 2022 at 15:18
  • $\begingroup$ @nbro No, that is not correct, even with VI there are priors and posteriors, what you learn is the transformation from one to another, its an approximation, but still it is Bayesian. $\endgroup$
    – Dr. Snoopy
    Dec 24, 2022 at 15:20
  • $\begingroup$ Yes, there are priors and posteriors, but that doesn't make it a Bayes rule. You do not apply the Bayes rule. It's Bayesian, yes, because you have priors and posterios, but you do not apply the Bayes rule $p(y \mid x) = \frac{p(x \mid y) p(y)}{ p(x)}$. $\endgroup$
    – nbro
    Dec 24, 2022 at 15:21
2
$\begingroup$

Probability is one way to solve classification problems. Still, there are other ways like clustering and K nearest neighbor approach where we tend to analyze the position of the current data point and its neighboring points to classify it. Also, in the decision tree classifier, information gain is the core concept used to classify.

Improve this answer
$\endgroup$
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .