0
$\begingroup$

I am currently training a Random Forest model on about 400 features per instance. The training ROC was about 0.95 which is pretty high I think. However, when visualizing the variable importance of the Random Forest model, I found that the most important features for the models are the availability flags.

That is, since I have some kind of irregular data, I aggregated data with statistical values such as mean, median, and so on. To note whether these values could be calculated in the last observation window, I added a binary flag. That is, if the flag is true, the statistical values are calculated, if the flag is false, the missing values were replaced with 0. Now these flags have a high variable importance, which means that there is some correlation that for class 1 the data is more likely to be present than for class 0.

However, this is not what the model is basing its prediction on, as it is just a percularity of the underlying data collection process. How do you normally deal with this?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If your model learns correlations using data, then those correlations are necessarily in the data. The only way for the model to not learn those correlations is to change your data.

You can either accept that there is a correlation with the flag variable, leading to a high result, or you can just remove the flag from the data you feed to your model. If you do not want the model to learn anything based on the flag, remove the flag altogether.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I see your point. However, removing the data would probably lead to the fact that correlations are learned with the replaced value of missing data. Is it a good idea to randomize these values using for replacement of missing value? $\endgroup$
    – Ai4l2s
    Feb 23 at 15:03
  • 1
    $\begingroup$ If the imputed missing data is correlating with the end result, your missing data might not be suited for imputation in the simple form in which you did. If there is structurally missing data, you might need to deal with it differently then you did. I recommend looking up 'different types of missing data' and what to do about it. $\endgroup$
    – Robin van Hoorn
    Feb 23 at 15:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .