Does the neural network calculate different relations between inputs automatically?

Question

Suppose you want to predict the price of some stock. Let's say you use the following features.

OpenPrice  
HighPrice
LowPrice
ClosePrice

Is it useful to create new features like the following ones?

BodySize = ClosePrice - OpenPrice  

or the size of the tail

TailUp = HighPrice - Max(OpenPrice, ClosePrice)  

Or we don't need to do that because we are adding noise and the neural network is going to calculate those values inside?

The case of the body size maybe is a bit different from the tail, because for the tail we need to use a non-linear function (the max operation). So maybe is it important to add the input when it is not a linear relationship between the other inputs not if it's linear?

Another example. Consider a box, with height $X$, width $Y$ and length $Z$.
And suppose the real important input is the volume, will the neural network discover that the correlation is $X * Y * Z$? Or we need to put the volume as input too?

Sorry if it's a dumb question but I'm trying to understand what is doing internally the neural network with the inputs, if it's finding (somehow) all the mathematically possible relations between all the inputs or we need to specify the relations between the inputs that we consider important (heuristically) for the problem to solve?

Answer 1

On paper, one expects a complex enough network to determine any complicated function of a limited number of inputs, given a large enough dataset. But in practice, there is no limit to the possible difficulty of the function to be learnt, and the datasets can be relatively small on occasion. In such cases - or arguably in general - it is definitely a good idea to define some combination of the inputs depending on some heuristics as you suggested. If you think some combination of inputs is an important variable by itself, you definitely should include it in your inputs.

We can visualize this situation in TensorFlow playground. Consider the circular pattern dataset on top left corner with some noise. You can use the default setting: $x_1$ and $x_2$ as inputs with 2 hidden layers with 4 and 2 neurons respectively. It should learn the pattern in less than 100 epochs. But if you reduce the number of neurons in the second layer to 2, it is not going to get as good as before. So, you are making the model more complicated to get the correct answer.

You can experiment and see that one needs at least one 3 neuron layer to get the correct classification from just $x_1$ and $x_2$. Now, if we examine the dataset, we see the circles so we know that instead of $x_1$ and $x_2$, we can try $x_1^2$ and $x_2^2$. This will learn perfectly without any hidden layers as the function is linear in these parameters. The lesson to be learnt here is that, our prior knowledge of the circle ($x_1^2 + x_2^2 = r^2$) and familiarity with the data helped us in getting a good result with a simpler model (smaller number of neurons), by using derived inputs.

Take the spiral data at the lower right corner for a more challenging problem. For this one, if you do not use any derived features, it is not likely to give you the correct result, even with several hidden layers. Keep in mind that every extra neuron is a potential source of overfitting, on top of being a computational burden.

Of course the problem here is overly simplified but I expect the situation to be more or less the same for any complicated problem. In practice, we do not have infinite datasets or infinite compute times and the model complexity is always a restriction, so if you have any reason to think some relation between your inputs is relevant for your final result, you definitely should include it by hand at the beginning.

Answer 2

The question is related to "feature extraction". Firstly, to tackle a regression problem like both the problems stated by you, you need to provide the neural network with the most relevant inputs that have a effect on the output. Eg. If you want your network to add x and y, you need to provide it training examples like input(x=1, y=3) and output (sum=4). This will make your network do exactly what you want.

But suppose you do not know whether what inputs should you train your network on, neural networks can take care of that too. Look at this example: Look at the first truth table. Notice that the output column is actually the first input column and the other two input columns are just random. Eventually, the network learns this relationship and provides the correct results. What we learnt: if you are unsure about which inputs should you choose for your network, just provide as many as possible, or as many combinations as possible. Neural networks excel in finding relationships in input data.

Next, talking of the volume problem, this is what I have been doing recently. It's actually an example of function approximation. Usually, the problem has multiple inputs and a single output (just like the addition problem), but the inverse is also possible. i.e., input : sum and output: x & y. This comes under one to many function mapping and multivariate regression. So YES, you need to provide the volume as input and x,y and z as outputs while training. The recommended configuration is one neuron in input layer, at least 6 hidden neurons and 3 neurons in output layer For magical results, you can use a deeper neural network rather than the shallow one suggested by me. But remember, neural networks have been proved to be **Universal Approximators*