How to introduce variation between instances of a neural network?

Question

What are some good ways to introduce variation between instances of a neural network?

I've heard about training each instance on different data, with the same data in a different order, through bagging, by randomising each instance's initial weights, or by altering the structure of the network (e.g. changing the activation functions or the number of nodes in each of the hidden layers). Are there any other common methods that I've missed and when would one use one type of method over another?

Context:
I'm writing a classification system built on a neural network. I've trained it as much as I can on the training data that I have, and am now trying to improve its performance beyond that. I've tested a few different ways to prevent over-fitting and ways to improve reliability and performance, and now I'd like to experiment with ensembling. For this I need multiple network instances with slight variations compared to each other, hence my question.

Answer 1

(I'm not a NPL expert, but I'll try to provide some ideas to try.)

As a general background, consider that bagging is a technique used to reduce the variance of the ensemble model therefore allowing the ensemble to generalize better than the single (e.g. neural-net) model: this is achieved by simply averaging predictions. In order to have this working you need $N$ models (even of different kinds, e.g. RF, NN, SVM, ...) that make independent errors, i.e. each of them is wrong in a different way (uncorrelated) compared to the other models.

• In Random Forests (RFs) and Extra Trees this is generally done by subsampling the set of features, and even the data samples.

Now, regarding neural networks you can:

• Change the weight initialization (e.g. from uniform to normal or vice-versa, and even the kind like Glorot or He init) to introduce initial randomness in each network of the ensemble.
• Load the data in a different order, or more generally train the network by setting a different random seed, so that all the randomization is a function of that seed. Having different random seeds for each network may help them achieve a different local minima.
• Try dropout: you should do this on one single net, so that your ensemble is actually a single model. The reason is that during training dropout randomly zeroes some units, resulting in a sort of multitude of different models at each forward pass. After training the final model would be an "average" of those. In fact, dropout is also motivated by being a sort of implicit way to create an ensemble. But in principle, you can try creating an ensemble from $N$ networks with dropout initialized in different ways (i.e. different seed, and also drop probability may help) - but I'm not so sure if it's work properly.
• Specifically for text you can try, at test time, to randomly mask some portion of the text, do this $N$ times (differently!) and predict each masked phrase with the same network, then average the predictions. A similar thing is done from images, although by random cropping.
• Lastly, there are general ensembling techniques for NNs like snapshot ensembling (use cyclic lr schedule, at each cycle save the weights, after training average them all), Polyak average (keep a copy of the weights that is a running average updated during training), and the stochastic weight averaging (SWA).