I have a simple neural network for a binary classification.

Input features include: age, sex, economic_situation, illness, disability, etc.

Output is simply 1 and 0

I would like to order the features for each input from greatest to least impact it had on the classification.

An example answer could look like this:

Classification: 1
1) illness
2) economic_situation
3) disability
4) sex
5) age

Another example:

Classification: 0 
1) economic_situation
2) age
3) disability
4) sex
5) illness

Two popular methods I’ve seen done:

1) For each feature, remove it and run the model and see the impact it has on the result. The idea is that the larger the impact, the more pertinent it was to the result.

2) Look at the gradients magnitude $|\nabla_f {y} |$. You can either look at the raw gradient or look at the guided back-propagation which is just the back props product rule, but you only look at when the nodes positively help trigger a neuron by taking only the positive gradients at each step.

There’s probably also more methods. Hope this helped.

  • $\begingroup$ We don't get all of the (nonlinear) interactions in vanilla linear regression, but #1 is how we do parameter inference in statistics! (At least it's a way.) $\endgroup$
    – Dave
    Jul 29 '20 at 17:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.