How do I select the relevant features of the data?

Question

Recently I was working on a problem to do some cost analysis of my expenditure for some particular resource. I usually make some manual decisions from the analysis and plan accordingly.

I have a big data set in excel format and with hundreds of columns, defining the use of the resource in various time frames and types(other various detailed use). I also have information about my previous 4 years of data and actual resource usage and cost incurred accordingly.

I was hoping to train a NN to predict my cost beforehand and plan even before I can manually do the cost analysis.

But the biggest problem I'm facing is the need to identify the features for such analysis. I was hoping there is some way to identify the features from the data set.

PS - I have idea about PCA and some other feature set reduction techniques, what I'm looking at is the way to identify them in the first place.

Answer 1

Since you have all your data in a table, a relatively simple thing to do is to consider each column independently, and then seeing if the output variable (cost incurred) has a correlation to that.

If the column has no (or very low correlation) with the output variable, then consider it to be not important. The ones that make the cut are then considered further.

This is obviously not very different from how a decision tree algorithm would work (such as ID3).

Answer 2

there is no hard-and-fast-rule for feature selection , you have to manually examine the dataset and try different techniques for feature engineering . And there is no rule that you should apply neural networks for this , neural networks are time-consuming to train , instead you can experiment with decision tree based methods(random forests ) since your data is anyway in tabular structure.

Answer 3

That's a great question and probably one of the most difficult tasks on ML.

You do have a few options:

1. You can use weighting algorithms (e.g. Chi-squared) to understand which features are contributing most to your output
2. You can use other ML algorithms to classify whether a feature is contributing to your predictions or not
3. You may use other ML algorithms (other than NN) that inherently provide you with feature weightings (e.g. Random Forest)

Hope that helps

Answer 4

It is wise to consider not just the correlation of resource engagement with cost, but also the return on the cost of resource engagement. The typical challenge is that those returns are almost always cumulative or delayed. A case of accumulation is when the resource is the continuous tuning or improvement of a process the absence of which slows the generation of revenue. A case of delay is when research resources incur costs without revenue impact for a period of time but the revenue generation that begins if the research delivers productive results may be a substantial factor above the total cost of the results delivered.

The reason expense data by itself can lead to maladaptive network learning is because a network that is trained to reduce, for instance, marketing expenses will zero them. That would usually cause a decreasing sales lead trend until the business folds. Without including the returns in the training information, no useful learning may occur.

A basic MLP (multi-layer perceptron) will not learn the temporal characteristics of the data, the accumulation and delay aspects. You will need a stateful network. The most consistently successful network type for this kind of learning as of this writing is the LSTM (long short term memory) network type or one of its derivative variants. Revenue and balance data must be used in conjunction with expense data to train the network to predict business results for any given sequence of proposed resource engagements (fully detailed budgetary plan).

The loss function must properly balance sort term with medium and long term financial objectives. Negative available cash should produce a pronounced increase in the loss function so that such avoidance of basic risks to reputation and the cost of credit is learned.

Which columns in your data have strong correlations with return on investment is difficult to determine in advance. You can immediately exclude columns that conform to any one of the following criteria.

• Always empty
• Other constants, those that have the same value for every row
• Those that can always be derived from other columns

The data can be reduced in other ways

• Fully describing data by characterizing trends in simple ways
• Using indices to specify long strings with 100% accuracy by assigning each string a number
• Compression
• Otherwise reducing redundancy in the data

RBMs (restricted Boltzmann machines) can extract features from the data and PCAs can illuminate the low information content columns, but the significance of the columns in terms of their correlation with revenue will not be identified using these devices in their basic form.