It is wise to consider not just the correlation of resource engagement with cost, but also the return on the cost of resource engagement. The typical challenge is that those returns are almost always cumulative or delayed. A case of accumulation is when the resource is the continuous tuning or improvement of a process the absence of which slows the generation of revenue. A case of delay is when research resources incur costs without revenue impact for a period of time but the revenue generation that begins if the research delivers productive results may be a substantial factor above the total cost of the results delivered.
The reason expense data by itself can lead to maladaptive network learning is because a network that is trained to reduce, for instance, marketing expenses will zero them. That would usually cause a decreasing sales lead trend until the business folds. Without including the returns in the training information, no useful learning may occur.
A basic MLP (multi-layer perceptron) will not learn the temporal characteristics of the data, the accumulation and delay aspects. You will need a stateful network. The most consistently successful network type for this kind of learning as of this writing is the LSTM (long short term memory) network type or one of its derivative variants. Revenue and balance data must be used in conjunction with expense data to train the network to predict business results for any given sequence of proposed resource engagements (fully detailed budgetary plan).
The loss function must properly balance sort term with medium and long term financial objectives. Negative available cash should produce a pronounced increase in the loss function so that such avoidance of basic risks to reputation and the cost of credit is learned.
Which columns in your data have strong correlations with return on investment is difficult to determine in advance. You can immediately exclude columns that conform to any one of the following criteria.
- Always empty
- Other constants, those that have the same value for every row
- Those that can always be derived from other columns
The data can be reduced in other ways
- Fully describing data by characterizing trends in simple ways
- Using indices to specify long strings with 100% accuracy by assigning each string a number
- Otherwise reducing redundancy in the data
RBMs (restricted Boltzmann machines) can extract features from the data and PCAs can illuminate the low information content columns, but the significance of the columns in terms of their correlation with revenue will not be identified using these devices in their basic form.