I understand the intuition behind stacking models in machine learning, but even after thorough cross-validation scheme models seem to overfit. Most of the models I have seen in kaggle forums are large ensembles, but seem to overfit very little.
$\begingroup$ Have you tried training each of the stacking models on separate training data? $\endgroup$– robitJun 29, 2018 at 9:21
$\begingroup$ yeah , separate models are perfect , but they overfit a lot when stacked , $\endgroup$– thecomplexitytheoristJun 29, 2018 at 9:35
$\begingroup$ It seems weird. I trained a GBDT+LR model: I split training dataset into two subsets A and B. Then, I trained GBDT with A, and LR with B. The stacked model works well. Besides, what models are you training? $\endgroup$– robitJun 30, 2018 at 2:14
$\begingroup$ random forests and knn $\endgroup$– thecomplexitytheoristJun 30, 2018 at 5:04
$\begingroup$ @thecomplexitytheorist, please may you describe the CV scheme that you are using. For example, are you using the scheme given on the Kaggle blog? In principle that can lead to overfitting (see the 'Stacked Model Hyper Parameter Tuning' section), though I've never experienced it $\endgroup$– BenJun 30, 2018 at 13:24
The effectiveness of dividing training data and piping divisions into networks for independent training, although possibly an effective workaround for specific cases, is not indicative of a robust solution to fitting excellence across a wide range of input data sets.
As suggested in the comment by varshaneya, over-fitting can be a result of unsatisfactory regularization meta-parameterization, as in a poor setting of the λ regularization parameter in a StackGAN. All meta-parameters used to tune a stacked architectures should be scrutinized to determine whether its setting could lead to over-fitting. A few can be eliminated up front. For instance too high a learning rate at any level of any of the networks in the design can reduce convergence probability, but is not a likely cause for over-fitting.
H. Hutson, S. Geva, and P. Cimiano wrote, in their 2017 submittal to the 13th NTCIR Conference on Evaluation of Information Access Technologies, "Ensemble methods in machine learning involve the combination of multiple classifiers via a variety of methods such as bagging (averaging or voting), boosting, and stacking, to increase performance and reduce over-fitting." Yet bagging has not produced robust responses to differing data sets in our experience, even when normalized, filtered to reduce noise levels, and redundancy is limited.
Zhi-Hua Zhou and Ji Feng [National Key Laboratory for Novel Software Technology, Nanjing University, China, indicated] wrote, "To reduce the risk of over-fitting, class vector produced by each forest is generated by k-fold cross validation." Reading their paper, Deep Forest, may give you some causes to evaluate.
Over-fitting is usually the application of too sophisticated a model to which data is fit. In the world of activated networks, excessive sophistication can be as simple as an excessive number of network layers in one or more of the stacked networks.
Feature extraction up front may be needed to remove complexity from the input which is not only unnecessary but counterproductive to generalization and thus the generation of useful output.