0
$\begingroup$

We have our training set and our test set. When we scale our data we "fit" the scaler transform to the training set and then we scale both the training set and test set using this scaler object. Using splitting and cross-validation techniques, one can use the training set as training and validation. Finally, reporting on the test set.

Now, if I want to use a model in a real-life environment, it's common to use the entire dataset (training and test) to train our already optimized model to obtain a final ready for the production model.

My question is regarding scaling. Should we fit the scaler to the entire set and then scale? Or can we simply append the scaled training set and scaled test set (both have been scaled using the training set's scaling parameters)?

I am making use of sklearn.preprocessing.PowerTransformer. Using "Yeo-Johnson's" power transform and also standardizing the data.

$\endgroup$
0
0
$\begingroup$

The short answer is yes.

When you merge the test set into the train set, you try to squeeze available data till the last drop. The cons and pros of this approach have been considered in other questions in the network 1, 2. But if you decided to go for it, there is no point to not use the whole dataset for the scaling transformation, as the trend "more data leads (generally) to better models" is valid to the scaling to the same extent it's valid for the model itself.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.