Should I allow NN to infer relationships of inputs?

Question

This question is assuming a sequential, deep neural network

Given some features [X1, X2, ... Xn], I'm trying to predict some value Y.

The raw data available to me contains feature X1 and feature X2. Say that I know there is an effect on Y based on the ratio of the two features, i.e. X1 / X2.

Should I add a new feature, mathematically defined as the ratio of the two features? I haven't been able to locate any literature which begins to describe the necessity or warnings of this.

Instinctly I'm worried about the following:

• Overfitting and the need for excessive regularization, due to duplicate information in the feature set
• Exponentially growing number of features, since defining a ratio between each feature may be necessary

However, I also recognize that certain relationships are impossible to be defined by a deep neural network (i.e. logic gates, exponential relationships, etc), so when would this sort of "relationship defining" be necessary? For example, if an exponential relationship is known to exist?

Answer 1

You are refering to the first and very important step in a machine learning process called data preprocessing. Refering to this article, inside data preprocessing there are many smaller processes that deal with features: feature extraction, feature selection, feature aggregration and feature encoding to name a few.

The idea of creating new features out of raw features is not new, but rather a well known concept in machine learning called feature extraction. Suppose you decide to create the new $X_1 / X_2$ feature and add it to the raw set of features ${X_1, X_2}$, because you are completely sure that it has an effect on $Y$, then that is a perfectly reasonable example of feature extraction.

Refering to the article above on feature extraction, notice that nowadays there is some growing consensus that when using deep neural networks, the first hidden layers of the network can serve the purpose of automatic feature extraction, without you manually adding the new features to the raw set of features. This premise stems from the fact that deep neural networks are essentially non-linear function approximators, so the first hidden layers can approximate any feature extraction that you do manually.

You can understand that in order to do automatic feature extraction, you might need more hidden layers and more neurons in each layer, to increase the computing power of your model. There is also the downside that you can not know for sure what is happening in the first hidden layers, if it is doing feature extraction the way you intended or not.

In your case, my advice would be to do the manual addition of $X_1 / X_2$ in the set of features, if you are only adding one such fraction or a handful. Also, if you are completely sure there is a correlation between those fractions and $Y$. If on the other hand, you are adding more than a handful of such fractions, then it does not make sense to add them all. It is hard to be completely sure that all the fractions have an effect on $Y$. It would also exhaust the memory space, by having a lot of features on the input layer. In that case I would suggest that you skip manual feature extraction, increase the number of hidden layers and let the network do its auto-magic.

Answer 2

Ideal advise is to feed the raw data to the neural networks to let neural networks make its own inference

Considering you have expert knowledge that $X1/X2$ has effect on $Y$ , here the new feature ($X1/X2$) is referred to as a derived feature

However, there are few advantages which can help you consider of using derived features like $X1/X2$ in your case

• Being a domain expert or SME choosing X1/X2 as an important feature, you can ideally accelerate the training process
• Highly advantageous when you are short on CPU time
• Most neural networks calculate sums fed through the activation functions, estimating the ratio or the multiplication requires lot of neurons especially if $X1/X2$ is an important feature
• You can use feature pruning techniques to eliminate redundant features
• One major drawback of neural network is often considered to be a black box model that’s the approximation of neural networks doesn’t give any insight of the form of function f. Use make use of feature selection algorithms to pare down feature space.
• With large number of parameters in model also increasing the risk of overfitting the network. You can however overcome this with good regularisation methods. You can also use feature pruning to avoid overfitting when you are having limited data.

I found a literature in the field of medicine that deals with deriving features based on expertise accurately identified the required states.

Heart rate variability-derived features based on deep neural network for distinguishing different anaesthesia states

The incorporation of four HRV-derived features in the time and frequency domain and a deep neural network could accurately distinguish between different anaesthesia states.

Answer 3

Model and Objective function are playing together. If you can have an objective function that somehow exclude this relation that you have in your mind, then the model can focus on learning to predict based on other information. Then you have trained the model, but if your downstream task should consider that at the end, you could manually add this relation and apply the relation you have in your mind at the end. That's my philosophical answer, but the question is how do you want to implement it?‌‌‌‌‌‌‌‌‌‌ It depends on details of the project, and this approach might not be feasible, but I guess your concern is real. The model is inclined to learn the shortest path to the answer, and if you provide it for it, then it will use it. If I were you, I might try to first normalize the features so they'd have similar effects. I don't know how this would be implemented in your problem though.