How to improve neural network training against a large data set of points with varying magnitude

Question

I am currently using TensorFlow and have simply been trying to train a neural network directly against a large continuous data set, e.g. $y = [0.014, 1.545, 10.232, 0.948, ...]$ corresponding to different points in time. The loss function in the fully connected neural network (input layer: 3 nodes, 8 inner layers: 20 nodes each, output layer: 1 node) is just the squared error between my prediction and the actual continuous data. It appears the neural network is able to learn the high magnitude data points relatively well (e.g. Figure 1 at time = 0.4422). But the smaller magnitude data points (e.g. Figure 2 at time = 1.1256) are quite poorly learned without any sharpness and I want to improve this. I've tried experimenting with different optimizers (e.g. mini-batch with Adam, full batch with L-BFGS), compared reduce_mean and reduce_sum, normalized the data in different ways (e.g. median, subtract the sample mean and divide by the standard deviation, divide the squared loss term by the actual data), and attempted to simply make the neural network deeper and train for a very long period of time (e.g. 7+ days). But after approximately 24 hours of training and the aforementioned tricks, I am not seeing any significant improvements in predicted outputs especially for the small magnitude data points.

---

Figure 1

---

Figure 2

---

Therefore, do you have any recommendations on how to improve training particularly when there are different data points of varying magnitude I am trying to learn? I believe this is a related question, but any explicit examples of implementations or techniques to handle varying orders of magnitude within a single large data set would be greatly appreciated.

Answer 1

I think there is no special method for training the neural network in large datasets. But I can add some suggestion for you:

1) Use the convolutional neural network for this dataset.

2) You can use huber loss instead of squared loss and see what happens.

3) See if you have enough small magnitude training data.

Also, please define your problem in more detail (like what those images represent and what you want to predict, what are the features etc).

Answer 2

One option is normalizing your data. In particular, min-max feature scaling to bring all values into the range [0,1] is particularly useful with gradient descent.

Answer 3

It might be that the large labels dominate the loss value, so the model pays more attention to them. You could use an L1 loss rather than an L2 loss, such that the predictions get pulled towards each label equally rather than being pulled more strongly when the label is further away from the prediction.

There may also be a data imbalance, i.e. you train on more large labels than small labels. This would also cause the model to pay more attention to becoming good at predicting the large labels. If this is the case, you can either collect more small labels or train more often on the small labels you have.

Another possibility is that your labels are not evenly distributed. For example, the gaps between large data points may be larger than between small data points, so the small data points all look similar to the model. You can plot a histogram of your labels to find out and then transform your labels (e.g. using log) to make space them out more evenly.

The general rules for training artificial neural networks apply. For example, normalize your inputs and outputs by subtracting the mean and divide them by their standard deviation, estimated across the training set. It also seem that your network is likely too deep and and not wide enough. I'd try 4x100 rather than 8x20 and add a small amount of weight decay when you see overfitting.

Answer 4

Probably you could:

• Normalize your dataset
• Add Regularization