# permutation importance shuffling

When calculating permutation importance for a certain feature, that feature is shuffled randomly and predictions with the shuffled feature are compared to predictions with the feature in its original order.

What are the advantages of this over filling the feature up with a certain value, randomly generated values, deleting the feature and retraining + predicting, and other possible alternatives?

Note: all the alternatives that I mentioned like filling the feature with randomly generated values would be used to predict several values and then those values would be compared with the standard ones

• Could you please clarify whether you are asking "Why use permutation importance?", or "Why use random shuffling to measure permutation importance?". Some of your suggested alternatives have nothing to do with measuring anything about permutations, and it is not clear whether that is because you have misunderstood what permutation importance is measuring, or whether you are questioning why someone might be interested in it. Aug 7 at 10:59
• @NeilSlater it is likely because I have misunderstood and am attempting to clarify with this question. Aug 7 at 17:54

Shuffling guarantees you that the noisy value still comes from the same distribution, thus if it comes from the distribution and doesn't create a lot of change then $$D_{KL}(p(y|x)||p(y))$$ it's very small, which means that the two distributions are similar and that $$x$$ is irrelevant