Do correlations matter when building neural networks?

Question

I am new to working with neural networks. However, I have built some linear regression models in the past. My question is, is it worth looking for features with a correlation to my target variable as I would normally do in a linear regression or is it better to feed the neural network with all the data I have?

Assuming that the data I have is all related to my target variable of course. I am working with this dataset and building a neural network regressor for it.

https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DL0101EN/labs/data/concrete_data.csv

Here is a snippet of the data. The target variable is the concrete strength rate given a certain combination of materials for that concrete sample.

I greatly appreciate any tips and explanations. I excuse me if this is too noob of a question but unfortunately I did not find any info about it on google. Thanks again!

Answer 1

If there is some correlation between features, that is what the network will ideally find out on its own and learn to utilize. So, in general, don't take correlated samples or features out of the training loop only because they look correlated. After all, they could convey a lot of valuable information.

When it comes to correlation between data samples during training, this correlation is commonly broken up by training a network on randomly selected mini-batches of training data samples. So, you randomly sample e.g. 16 or 32 (or so) training examples based on which you apply a single update of the weights using some Stochastic Gradient Descent variant. Since the members of a mini-batch are sampled at random, chances for finding highly correlated training samples in some mini-batch shall be sufficiently minimized in order not to negatively affect the training outcome.

Having said that, if you are concerned about overfitting of your model or weights that would overly weight just a small subset of all available input features, you could try applying regularization techniques like L1 (encouraging sparse representations) or L2 (encouraging low weights in general) regularization or dropout. In your particular case, since the main concern is an excessive contribution of only a small set of input features, L2 shall yield better results (avoiding excessively large weights that would be required to excessively much weight just a small number of features).

Besides that: Commonly, you split your training dataset into 3 parts:

1. Data used for fitting the model (actual training data)
2. Data used for assessing the training progress & possibly for determining when to apply early stopping (validation data)
3. Test data used to assess the performance of the system after all training & intermediate testing is done

The final evaluation on the test dataset shall reveal then the generalization ability of your trained model to novel data.

So, with regularization in place during training and relatively low error rates on the validation and test datasets, you are pretty much save even without checking for correlated data beforehand. Only when you really struggle decreasing the validation loss, it might be worth to further inspect what exactly is going wrong in terms of correlations and such.