I am currently training a Random Forest model on about 400 features per instance. The training ROC was about 0.95 which is pretty high I think. However, when visualizing the variable importance of the Random Forest model, I found that the most important features for the models are the availability flags.
That is, since I have some kind of irregular data, I aggregated data with statistical values such as mean, median, and so on. To note whether these values could be calculated in the last observation window, I added a binary flag. That is, if the flag is true, the statistical values are calculated, if the flag is false, the missing values were replaced with 0. Now these flags have a high variable importance, which means that there is some correlation that for class 1 the data is more likely to be present than for class 0.
However, this is not what the model is basing its prediction on, as it is just a percularity of the underlying data collection process. How do you normally deal with this?