# What is the impact of scaling the features on the performance of the model?

I am trying to generate a model that uses several physicochemical properties of a molecule (including number of atoms, number of rings, volume, etc.) to predict a numeric value $$Y$$. I would like to use PLS Regression, and I understand that standardization is very important here. I am programming in Python, using scikit-learn.

The type and range for the features varies. Some are int64 while others are floating point numbers. Some features generally have small (positive or negative) values, while others have a very large value. I have tried using various scalers (e.g. standard scaler, normalize, min-max scaler, etc.). Yet, the R2/Q2 are still low.

I have a few questions:

1. Is it possible that by scaling, some of the very important features lose their significance, and thus contribute less to explaining the variance of the response variable?

2. If yes, if I identify some important features (by expert knowledge), is it OK to scale other features but those? Or scale the important features only?

3. Some of the features, although not always correlated, have values that are in a similar range (e.g. 100-400), compared to others (e.g. -1 to 10). Is it possible to scale only a specific group of features that are within the same range?

In general, algorithms that exploit distances or similarities (e.g. in the form of scalar product) between data samples, such as k-NN and SVM, are sensitive to feature transformations. We do feature scaling to make our model robust to outliers and make an initial impact of every feature on the model will be roughly similar

Graphical-model based classifiers, such as Fisher LDA or Naive Bayes, as well as Decision trees and Tree-based ensemble methods (RF, XGB) are invariant to feature scaling, but, still, it might be a good idea to rescale/standardize your data.

1. You should explore your data more carefully, find the outliers, apply transformation if needed.

2. Not sure if it is a good idea

3. You can apply different preprocessing techniques like MinMaxScaller, Rank, Log transform, Extracting square root, StandartScaller and etc.

Project Statement

From the text of the question it appears that the project is to produce a model of compounds and their properties from validated data so that those properties can be predicted for compounds not yet characterized. The properties include these.

• Number of atoms per molecule
• Number of cycles (rings) per molecule
• Specific volume at some equilibrium temperature and pressure

The predictability of such properties may be possible within some boundaries of reliability and accuracy. For instance, just as an example, such a system may produce three significant digits of accuracy for one of the properties for 99.6% of the compounds to which intelligent prediction is applied.

The question author would like to use PLS regression and is aware that selection of type (int64, float, ...) and the standardization of the data is important. The author tried scaling automation, including those below, but is concerned that the R2/Q2 ratio is low, which is a reasonable concern.

• standard scaler
• normalize
• minmax scaler

Scaling, Standardizing, and Normalization

Just as compounds can be characterized, the distribution of data for a particular property across a large set of compounds can also be characterized at a higher level of abstraction.

The most obvious is the domain of the features used as input to the learning process. (These same dimensions of the data, which have domains during when they are inputs to the learning process, will have ranges when they are the outputs of the trained network.)

The domain of training data dimensions (both features and labels) can be examine for their characteristics.

• Absolute minimum in the data
• Theoretical minimum
• Absolute maximum in the data
• Theoretical maximum
• Mean(s)
• Standard deviation(s)
• General type of distribution (Gaussian, Poisson, inverse exponential, ...)
• Skew(s)

Combined with the granularity and expected accuracy of each dimension, the type of variable to represent the data for each dimension can be selected.

• Integer
• Fixed point
• Floating point
• String
• Recursive structure

The size of the variable type and the normalization of the data is not something easy to automate. In fact, experts tend to put it off until after the distribution is well understood, perhaps along with the correlation of input dimensions to understand redundancy versus uniqueness in the data. The goal, when preparing input for learning systems in general, is to not waste bits with redundancy or superfluousness, either across the training examples or across the dimensions of each.

An example of this across the entire training set would be a user column or a binary flag that indicates whether the example is a molecule. The user is irrelevant to the training and could actually distort the results, and the entire training set is for molecules, so that flag is redundant across the domain of example indices.

An example of this between features is molecular weight in combination with the molecular formula. The atomic weights and their quantities for each element within the molecule is within the formula. Molecular weight can be calculated from this information. If the formula is properly represented in the input vector, the addition of an input for molecular weight may actually be counterproductive.

With a proper encoding and normalization of data, it may be possible to predict crystalline tendencies, thermal properties, electrical properties, and phase change surface in PVT space. It would take experimentation to determine the feasibility of those.

In general, the best scenario is when the range of the variable is a superset of the expected long-term domain of the values, but by a minimal margin on both sides and the distribution of values is substantially normal. There is more information about normalization here: Gradient Descent Feature Scaling.

Specific Questions

These sub-questions are specifically listed.

Is it possible that by scaling, some of the very important features lose their significance, and thus contribute less to explaining the variance of the response variable?

Only if the scaling is done poorly, not according to the above principles. If, in the preparation of input, information is lost, it is not available to the learning system. Don't do that.

If yes, if I identify some important features (by expert knowledge), is it OK to scale other features but those? Or scale the important features only?

Use the expert knowledge. Notice the difference between the min and max in the data and the theoretical min and max. Take the union of both of these to ensure nothing falls outside the range of the standardization and choice of data type for input. With regard to what is and is not scaled. Don't scale if it is not needed to utilize the above principles, such as a flag that fits perfectly into a Boolean input. Otherwise, scale.

Only use the scaling automation after you've checked the data and know what the automation will do in general before you use it. Scalars are good for automating calculations, not for making choices about data normalization. If and when computers become smarter than us with complex conceptual decision making, then we can delegate cognitive tasks to them.

Some of the features, although not always correlated, have values that are in a similar range (e.g. 100-400), compared to others (e.g. -1 to 10). Is it possible to scale only a specific group of features that are within the same range?

If some dimensions are similar in distribution and extent (min and max), the findings and the methodology applied to one dimension can be reused by the others of similar distributions and extents.

Footnotes

[1] It is not clear why the number of atoms per molecule would require prediction, since a simple parsing of the compound's formula should provide that number.