1
$\begingroup$

Suppose we have a data set $X$ that is split as $X_{\text{train}}$, $X_{\text{val}}$ and $X_{\text{test}}$ and the outcome variable is binary. Let's say we train three different models (logistic regression, random forest, and a support vector machine) using $X_{\text{train}}$. We then get predictions for $X_{\text{val}}$ using each of the three models.

In stacking, is it correct to say that we train a logistic regression model on a data set of dimension $|X_{\text{val}}| \times 3$ with the predicted values and actual values of the validation set? This logistic regression model is then used to predict outcomes for data in $X_{\text{test}}$?

$\endgroup$
  • $\begingroup$ By stacking, do you mean ensemble learning? $\endgroup$ – nbro Apr 28 at 12:25
  • $\begingroup$ Do you mind if I edit the title of your question to be a bit more general so it is more clear the context of the post? $\endgroup$ – Hanzy Apr 28 at 13:59
0
$\begingroup$

The predictions of each of your initial models will become a feature to feed the meta learner. If you use $n$ initial models, then for each example you will feed the meta learner $n$ features, each feature being a prediction from one of the initial models.

Note that this doesn’t mean the size of your dataset increases. Instead, each member of $X_{val}$ will be represented by $n$ features, with the $k$th feature being equal to the initial prediction of the $k$th initial predictor.

This medium link recommends a package that can help you build a pipeline.

Note: in a sense if you are caching the intermediate predictions of each initial predictor, you’ll end up with $n$ times as many data points as there are entries in $X_{val}$. But these are simply each going to be a feature of another data point, so the dataset hasn’t really increased in size.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.