How to fill NaNs in Cross-Validation?

Question

I have been searching this but did not find the answer, so sorry if this is a duplicated question.

I was working with cross-validation, where some doubts came to my mind, and I am not sure which is the correct answer.

Let's say I have a mixed dataset, with numerical and categorical features. I want to perform a K-Fold Cross-Validation with it, with a K=10. Some of these numerical features are missing, so I decided that I will replace those NaNs with the average of that feature.

My steps are the following ones:

1. Read the entire dataset
2. Perform One Hot Encoding to categorical features.
3. Divide my data into different folds. Let's say that I will use 90% for training, 10% for validating.
4. For every different combination of folds, I replace the missing values from the training and validating sets separately. This means, on one hand, I get the average of the missing values of the training part, and on the other hand the average of the missing values of the validating part.
5. Normalize the data of the training and validating sets between [0, 1] separately, as I did before.
6. Train the correspondant model.

So let's put a simple example of a dataset of 20 rows with N columns. Once I do steps 1 and 2, in the first iteration I will select the 18 first rows as a training set, and the last two rows as validating set. I fill the missing values of the 9 first rows with the average of those 18 rows. Then the same for the 2 last rows. Then, again, normalize in the same way, separately. And do this for every combination of folds.

I am doing it like this, because otherwise, from my understanding, is that you are training your model with biased data. You should not have access to the validation data, thus you should not be able to do the average with those numbers. Hence I am using only the numbers of the training part. If I do the average with the entire dataset, this will make my model overfitting.

I am not so sure about the normalization step, as I do not really think this will have the same impact. But here I do not really know...

Is this approach correct? Or should I do the average and normalization with the entire dataset? Why?

Answer 1

I would do the exact same thing as you are describing! One of the main reasons that you would want to do cross-validation is to prevent that your model is unable to generalize later. Therefore, you take out a random small subset which will be your new small validation set and do all the 'operations' on that which you are also doing on your training set. This way, you can check that these 'operations' are thus also generalizable (if they work for all different cross-validations).