# Can neural networks be used to find features importance?

I am wondering if I can use neural networks to find features importances in similar manner as it can be done for random forests or decision trees and if so, how to do it?

I would like to use it on tabular time series data (not images). The reason why I want to find importances on neural networks not on decision trees is that NNs are more complicated algorithms so using NNs might point out some correlations that are not seen by simple algorithms and I need to know what features are found to be more useful with that complicated correlations.

I am not sure if I made it clear enough, please let me know if I have to explain something more.

This should be possible, considering universal approximation theorem you should be able to build a ann that approximates features that gives the most likely best feature set for a different net to train on. I would us a rnn for with a softmax output layer that ranks features by performance.

You can find a good explanation of softmax here: https://developers.google.com/machine-learning/crash-course/multi-class-neural-networks/softmax basically it will assign probability values for each output node with all of these values adding up to 1.0

• Could you please explain the procedure in a bit more detailed way or give some reference link? – Makintosz Aug 13 '19 at 5:03
• I added a bit of info on softmax, what are you using for optimization? knowing that i may be able to provide a bit more insight into how i would set this up – nickw Aug 13 '19 at 21:36
• Thanks for more info! I am using tensorflow library, mostly with Adam optimizer (trying others as well) with custom learning rate. – Makintosz Aug 14 '19 at 7:22
• at a ten thousand foot level this could be done by deciding the amount of features or the min probability as a cut of and then creating a net with softmax output layer with a node for each feature, you will likely want to use a reinforcement learning optimization technique as you will need to evaluate if how well the features the net picks perform in order to optimize, you could use neuroevolution with a fitness function for this or i imagine other rl algos would work too. – nickw Aug 20 '19 at 14:17

There are multiple standard ways of feature selection, for example ranking features by information gain, that you could use, and then you can train the neural network on just those features.

However, let's assume you have trained a neural network on all of the features and now want to estimate their importance. One approach you could take is to perform a sensitivity analysis on the inputs: add random noise in a controlled fashion to different features and see what effect it has. If the training dataset has been centered (so each feature has zero mean) then you could set the inputs to zero (the "average" training example) and then perturb each feature in turn to see what the effect is. You could also fix a feature permanently to zero and then run your validation data through the network and see how accuracy changes. There should be no major effect for insignificant features, but important features being zeroed should lead to a decrease in accuracy. You can also do something like this when predicting specific examples: perturb the example's features and see how much the prediction changes. LIME does something like this to explain why a black box like a neural network makes the predictions it does.

It's possible. I've used the olden() function from NeuralNetTools. Please take a look at this example found online: http://blogs2.datall-analyse.nl/2016/02/19/rcode_variable_importance_neural_network/

• A bit more theory and proper explanation of potential solutions is warranted here. – hisairnessag3 Aug 12 '19 at 14:00
• can you elaborate down your answer for the op – quintumnia Aug 12 '19 at 14:40