How can an ensemble be more accurate than the best base classifier in that ensemble?

Question

BACKGROUND: Ensemble classifiers are said to reduce bias by taking an "average" of predictions of several base classifiers that comprise the ensemble. However, I am uncertain if this necessarily means that they can increase accuracy. My intuition tells me that the ensemble classifier should perform no better and possibly even worse than the best base classifier in the ensemble. This seems especially true for bagging approaches which use strong classifiers anyway. When you have a "star performer", it just doesn't seem to make intuitive sense to "dilute" its performance with subpar performers.

Nonetheless, from my novice-level reading, it seems that ensembles can be as good or possibly even better than all of the individual base classifiers, but I'm still not clear why.

QUESTION: How can an ensemble be more accurate than the best base classifier in that ensemble?

Answer 1

Lets consider binary classification.

Imagine you have an ensemble made up of $K$ models. Assume each model has exactly $51\%$ accuracy. Further assume each model's error is uncorrelated with each other model. The ensemble simply takes a majority vote.

For $K = 11$, what is the probability we get a single question right? Note: This problem can be simply modeled as flipping an unfair coin $K$ times and calculating the probability that one side will be the majority of flips.

$P(correct\ \geq\ 6)\ =\ \sum_{k=6}^{11}{\binom{11}{k}({0.51}^k)({0.49}^{11-k})= 0.53}$

What about when $K = 101$?

$P(correct\ \geq\ 51)\ =\ \sum_{k=51}^{101}{\binom{101}{k}({0.51}^k)({0.49}^{101-k})= 0.58}$

If you do the math you will find that as $K$ approaches $\infty$, the odds of the ensemble being correct asymptotically approaches $100\%$.

$\lim_{K\rightarrow\infty}{P\left(correct\ \geq\frac{K+1}{2}\right)}={lim}_{K\rightarrow\infty}\left(\sum_{k=\frac{K+1}{2}}^{K}{\binom{K}{k}({0.51}^k)({0.49}^{K-k}})\right)=1$

Now, we made 2 key assumptions:

1. Errors are uncorreleated between models
2. Each model has more than $50\%$ accuracy

In practice these assumptions often may not hold, especially the first one. But if they do, this simple probability analysis shows ensembles will be better