I read very often that Bayesian algorithms work well on small datasets. Why is that? I think it is because they might generalize more, but why is that?
See also Investigating the use of Bayesian networks for small dataset problems.
$\begingroup$
$\endgroup$
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The main reason should be that Bayesian algorithms naturally incorporate a form of regularisation (the prior), so they should be less prone to over-fitting the small dataset. Of course, the choice of the prior can affect your estimates.
You can view certain training regimes in machine learning and deep learning as an application of Bayesian statistics: for example, if you train a neural network with weight decay, this is equivalent to a MAP estimate.
I guess it might also be computationally easier to perform Bayesian inference with a smaller dataset (although I have never tried to show that this is true: my practical experience in this context is limited to Bayesian neural networks). Of course, the more data you have, in theory, the closer your estimates should be to the actual true values.
Here you have a similar question.