Is the final model scaling done on the full training set?

Question

We have our training set and our test set. When we scale our data we "fit" the scaler transform to the training set and then we scale both the training set and test set using this scaler object. Using splitting and cross-validation techniques, one can use the training set as training and validation. Finally, reporting on the test set.

Now, if I want to use a model in a real-life environment, it's common to use the entire dataset (training and test) to train our already optimized model to obtain a final ready for the production model.

My question is regarding scaling. Should we fit the scaler to the entire set and then scale? Or can we simply append the scaled training set and scaled test set (both have been scaled using the training set's scaling parameters)?

I am making use of sklearn.preprocessing.PowerTransformer. Using "Yeo-Johnson's" power transform and also standardizing the data.

Answer 1

The short answer is yes.

When you merge the test set into the train set, you try to squeeze available data till the last drop. The cons and pros of this approach have been considered in other questions in the network 1, 2. But if you decided to go for it, there is no point to not use the whole dataset for the scaling transformation, as the trend "more data leads (generally) to better models" is valid to the scaling to the same extent it's valid for the model itself.