Do we train a logistic regression model using a dataset that is 3 times bigger than the validation dataset?

Question

Suppose we have a data set $X$ that is split as $X_{\text{train}}$, $X_{\text{val}}$ and $X_{\text{test}}$ and the outcome variable is binary. Let's say we train three different models (logistic regression, random forest, and a support vector machine) using $X_{\text{train}}$. We then get predictions for $X_{\text{val}}$ using each of the three models.

In stacking, is it correct to say that we train a logistic regression model on a data set of dimension $|X_{\text{val}}| \times 3$ with the predicted values and actual values of the validation set? This logistic regression model is then used to predict outcomes for data in $X_{\text{test}}$?

Answer 1

The predictions of each of your initial models will become a feature to feed the meta learner. If you use $n$ initial models, then for each example you will feed the meta learner $n$ features, each feature being a prediction from one of the initial models.

Note that this doesn’t mean the size of your dataset increases. Instead, each member of $X_{val}$ will be represented by $n$ features, with the $k$th feature being equal to the initial prediction of the $k$th initial predictor.

This medium link recommends a package that can help you build a pipeline.

Note: in a sense if you are caching the intermediate predictions of each initial predictor, you’ll end up with $n$ times as many data points as there are entries in $X_{val}$. But these are simply each going to be a feature of another data point, so the dataset hasn’t really increased in size.