Normalization of possibly not fully representative data

Question

I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:

1. outliers, but I could fight them or use RobustScaler instead.
2. I am not sure that some features in my dataset include all possible ranges. Like I have max(feature_A) == 10, but with the data update, it could become 20. And if I'll preprocess data the same way I will get bad prediction results.
3. Some features do not have a limit at all and will only increase with time, like how many years patients were treated, for example. I could suppose that this value is !>100years, for example, but if my mean value is 10 years, it will squeeze feature values a lot.

My dataset is pretty large, like millions of observations, thus there is a pretty good chance that it is representative, though. But I am concerned with the small-time range, like all those observations are for the 2 years only, thus, some feature values (like how many years patients were treated) could still grow their bounds.

How should I handle this?

My concerns example:

import pandas as pd
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

#### like, initial state
df1 = pd.DataFrame({'A': [1, 2, 3, 4, 5], 'B': [10, 40, 60, 80, 100]})
""" output:
   A    B
0  1   10
1  2   40
2  3   60
3  4   80
4  5  100
"""

scaler.fit_transform(df1)
""" output:
array([[0.        , 0.        ],
       [0.25      , 0.33333333],
       [0.5       , 0.55555556],
       [0.75      , 0.77777778],
       [1.        , 1.        ]])
"""

#### new data arrived, while preprocessing is the same
df2 = pd.DataFrame({'A': [1, 2, 3, 4, 5, 10, 10], 'B': [10, 40, 60, 80, 100, 120, 140]})
""" output:
    A    B
0   1   10
1   2   40
2   3   60
3   4   80
4   5  100
5  10  120
6  10  140
"""

# now 5 in "A" scaled to 0.4 instead of 1, same in "B"
scaler.fit_transform(df2)
""" output:
array([[0.        , 0.        ],
       [0.11111111, 0.23076923],
       [0.22222222, 0.38461538],
       [0.33333333, 0.53846154],
       [0.44444444, 0.69230769],
       [1.        , 0.84615385],
       [1.        , 1.        ]])
"""

PS: I've duplicated this question in different communities (question in ai got most of views):

• https://stats.stackexchange.com/questions/523097/normalization-of-possibly-not-fully-representative-data
• https://datascience.stackexchange.com/questions/94095/normalization-of-possibly-not-fully-representative-data

Answer 1

A friend of mine answered on this question in different social media on different language, I'll post his answer here:

1. scaler should be saved in this case.

You do fit_transform in the example the second time you run it, but you should just transform. Scaler should be "fit" once on data train and not change it afterwards. Then you will get 5 in 1 in both cases. And when 10 appears, scaler will convert it to 2.

2. It's not a fact that this will break the model.

I think everything is clear here, but just in case: Let us consider the simplest case. Suppose there is a linear dependence y = 3 * x, we don't know about it, but there is a dataset:

X Y
1 3
2 6
3 9

If we assume that the model has the form y = a * x, in the process of learning it turns out that
a = 3 gives the best score and when suddenly x = 100 appears, it has no problem mapping to 300, even though 100 is strongly out of dataset (1,2,3).

3. You can check if this breaks the model now.

Suppose there is a chip that takes values 0, 1, 2, 3, 4, 5, 6, 7, 8 ... You are afraid that after some time samples with values greater than the current maximum (9, 10, 11, ...) will appear and the model will behave strangely on them. You can try to "simulate" this situation. Split dataset into train and test so samples with values from 0 to 6 will go into train, and samples with values from 7 to 8 will go into test. If accuracy doesn't fail at the test stage, then we can assume that when all the 0-8 ranks get into train, then the appearance of 9-11 won't break anything.

Also, there is answer for the second question: https://stackoverflow.com/questions/46744076/machine-learning-normalizing-features-with-no-theoretical-maximum-value