How to prevent overfitting in stacked models?

Question

I understand the intuition behind stacking models in machine learning, but even after thorough cross-validation scheme models seem to overfit. Most of the models I have seen in kaggle forums are large ensembles, but seem to overfit very little.

Answer 1

The effectiveness of dividing training data and piping divisions into networks for independent training, although possibly an effective workaround for specific cases, is not indicative of a robust solution to fitting excellence across a wide range of input data sets.

As suggested in the comment by varshaneya, over-fitting can be a result of unsatisfactory regularization meta-parameterization, as in a poor setting of the λ regularization parameter in a StackGAN. All meta-parameters used to tune a stacked architectures should be scrutinized to determine whether its setting could lead to over-fitting. A few can be eliminated up front. For instance too high a learning rate at any level of any of the networks in the design can reduce convergence probability, but is not a likely cause for over-fitting.

H. Hutson, S. Geva, and P. Cimiano wrote, in their 2017 submittal to the 13th NTCIR Conference on Evaluation of Information Access Technologies, "Ensemble methods in machine learning involve the combination of multiple classifiers via a variety of methods such as bagging (averaging or voting), boosting, and stacking, to increase performance and reduce over-fitting." Yet bagging has not produced robust responses to differing data sets in our experience, even when normalized, filtered to reduce noise levels, and redundancy is limited.

Zhi-Hua Zhou and Ji Feng [National Key Laboratory for Novel Software Technology, Nanjing University, China, indicated] wrote, "To reduce the risk of over-fitting, class vector produced by each forest is generated by k-fold cross validation." Reading their paper, Deep Forest, may give you some causes to evaluate.

Over-fitting is usually the application of too sophisticated a model to which data is fit. In the world of activated networks, excessive sophistication can be as simple as an excessive number of network layers in one or more of the stacked networks.

Feature extraction up front may be needed to remove complexity from the input which is not only unnecessary but counterproductive to generalization and thus the generation of useful output.