If there is some correlation between features, that is what the network will ideally find out on its own and learn to utilize. So, in general, don't take correlated samples or features out of the training loop only because they look correlated. After all, they could convey a lot of valuable information.
When it comes to correlation between data samples during training, this correlation is commonly broken up by training a network on randomly selected mini-batches of training data samples. So, you randomly sample e.g. 16 or 32 (or so) training examples based on which you apply a single update of the weights using some Stochastic Gradient Descent variant. Since the members of a mini-batch are sampled at random, chances for finding highly correlated training samples in some mini-batch shall be sufficiently minimized in order not to negatively affect the training outcome.
Having said that, if you are concerned about overfitting of your model or weights that would overly weight just a small subset of all available input features, you could try applying regularization techniques like L1 (encouraging sparse representations) or L2 (encouraging low weights in general) regularization or dropout.
In your particular case, since the main concern is an excessive contribution of only a small set of input features, L2 shall yield better results (avoiding excessively large weights that would be required to excessively much weight just a small number of features).
Besides that: Commonly, you split your training dataset into 3 parts:
- Data used for fitting the model (actual training data)
- Data used for assessing the training progress & possibly for determining when to apply early stopping (validation data)
- Test data used to assess the performance of the system after all training & intermediate testing is done
The final evaluation on the test dataset shall reveal then the generalization ability of your trained model to novel data.
So, with regularization in place during training and relatively low error rates on the validation and test datasets, you are pretty much save even without checking for correlated data beforehand. Only when you really struggle decreasing the validation loss, it might be worth to further inspect what exactly is going wrong in terms of correlations and such.