1
$\begingroup$

I am wondering if I can use neural networks to find features importances in similar manner as it can be done for random forests or decision trees and if so, how to do it?

I would like to use it on tabular time series data (not images). The reason why I want to find importances on neural networks not on decision trees is that NNs are more complicated algorithms so using NNs might point out some correlations that are not seen by simple algorithms and I need to know what features are found to be more useful with that complicated correlations.

I am not sure if I made it clear enough, please let me know if I have to explain something more.

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
2
$\begingroup$

This should be possible, considering universal approximation theorem you should be able to build a ann that approximates features that gives the most likely best feature set for a different net to train on. I would us a rnn for with a softmax output layer that ranks features by performance.

You can find a good explanation of softmax here: https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/softmax basically it will assign probability values for each output node with all of these values adding up to 1.0

Improve this answer
$\endgroup$
4
  • $\begingroup$ Could you please explain the procedure in a bit more detailed way or give some reference link? $\endgroup$
    – GKozinski
    Aug 13, 2019 at 5:03
  • $\begingroup$ I added a bit of info on softmax, what are you using for optimization? knowing that i may be able to provide a bit more insight into how i would set this up $\endgroup$
    – nickw
    Aug 13, 2019 at 21:36
  • $\begingroup$ Thanks for more info! I am using tensorflow library, mostly with Adam optimizer (trying others as well) with custom learning rate. $\endgroup$
    – GKozinski
    Aug 14, 2019 at 7:22
  • $\begingroup$ at a ten thousand foot level this could be done by deciding the amount of features or the min probability as a cut of and then creating a net with softmax output layer with a node for each feature, you will likely want to use a reinforcement learning optimization technique as you will need to evaluate if how well the features the net picks perform in order to optimize, you could use neuroevolution with a fitness function for this or i imagine other rl algos would work too. $\endgroup$
    – nickw
    Aug 20, 2019 at 14:17
0
$\begingroup$

There are multiple standard ways of feature selection, for example ranking features by information gain, that you could use, and then you can train the neural network on just those features.

However, let's assume you have trained a neural network on all of the features and now want to estimate their importance. One approach you could take is to perform a sensitivity analysis on the inputs: add random noise in a controlled fashion to different features and see what effect it has. If the training dataset has been centered (so each feature has zero mean) then you could set the inputs to zero (the "average" training example) and then perturb each feature in turn to see what the effect is. You could also fix a feature permanently to zero and then run your validation data through the network and see how accuracy changes. There should be no major effect for insignificant features, but important features being zeroed should lead to a decrease in accuracy. You can also do something like this when predicting specific examples: perturb the example's features and see how much the prediction changes. LIME does something like this to explain why a black box like a neural network makes the predictions it does.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .