I would like to apply the permutation feature importance technique to rank the features of a siamese network model that I trained. I am currently using this siamese network to perform some kind of classification, where I have many items coming from different categories, and I consider an item correcly classified if the network gives the highest similarity with another item from the same category. I would be interested to understand which features are determining the similarity for each class.
My thought was to actually calculate the permutation feature importances on my test data, but divided by class (so basically, if I have N features and M classes, I would get M different importances for each of the N features).
Does it make sense? Or are you aware of other suitable methods?
