0
$\begingroup$

When calculating permutation importance for a certain feature, that feature is shuffled randomly and predictions with the shuffled feature are compared to predictions with the feature in its original order.

What are the advantages of this over filling the feature up with a certain value, randomly generated values, deleting the feature and retraining + predicting, and other possible alternatives?

Note: all the alternatives that I mentioned like filling the feature with randomly generated values would be used to predict several values and then those values would be compared with the standard ones

Improve this question
$\endgroup$
2
  • $\begingroup$ Could you please clarify whether you are asking "Why use permutation importance?", or "Why use random shuffling to measure permutation importance?". Some of your suggested alternatives have nothing to do with measuring anything about permutations, and it is not clear whether that is because you have misunderstood what permutation importance is measuring, or whether you are questioning why someone might be interested in it. $\endgroup$
    – Neil Slater
    Aug 7 at 10:59
  • $\begingroup$ @NeilSlater it is likely because I have misunderstood and am attempting to clarify with this question. $\endgroup$
    – catasaurus
    Aug 7 at 17:54

1 Answer 1

Reset to default
2
$\begingroup$
  1. Distributional mismatch, the value that you pick might be out of distribution, and if inside the distribution, might be that it correspond to the correct one for some samples
  2. Again, distributional mismatch
  3. Might be expensive

Shuffling guarantees you that the noisy value still comes from the same distribution, thus if it comes from the distribution and doesn't create a lot of change then $D_{KL}(p(y|x)||p(y))$ it's very small, which means that the two distributions are similar and that $x$ is irrelevant

As for all ML algorithms, you have tradeoffs, and there is no specific reason why you should use a method over the other, if two are theoretically sound, then it's up to your experience and use case to use one over the other

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .