Creating a classifier for simpler classifiers trained on few training samples

Question

Suppose I have a classification problem with a stream of training-samples constantly arriving over time. I cannot keep all training-samples in memory, but I still want to train a classifier that will have the "wisdom" of all samples, and additionally, I want the classifier to become better whenever it gets new samples.

I thought of the following idea. Suppose we have enough memory to keep 100 samples. Then, for each run of 100 samples, we will train a different sub-classifier. We will have a meta-classifier that will classify based on voting between all existing sub-classifiers. Over time, we will have more and more sub-classifiers, so hopefully the meta-classifier will improve with time - it will have a "wisdom of the crowds" effect.

Has this method been tried before? Specifically, has it been tried in a deep-learning sequence-classification setting?

Answer 1

The voting technique that you described is called Ensemble Learning and its improvement over time is guaranteed if each classifier is at least a little better than random.