In NN, as iterations of Gradient descent increases, the accuracy of Test/CV set decreases. how can i resolve this?

Question

As mentioned in the title I'm using 300 Dataset example with 500 feature as an input.

As I'm training the dataset, I found something peculiar. Please look at the data shown below.

Iteration 5000 | Cost: 2.084241e-01

Training Set Accuracy: 100.000000

CV Set Accuracy: 85.000000

Test Set Accuracy: 97.500000

Iteration 3000 | Cost: 2.084241e-01

Training Set Accuracy: 98.958333

CV Set Accuracy: 85.000000

Test Set Accuracy: 97.500000

Iteration 1000 | Cost: 4.017322e-01

Training Set Accuracy: 96.875000

CV Set Accuracy: 85.000000

Test Set Accuracy: 97.500000

Iteration 500 | Cost: 5.515852e-01

Training Set Accuracy: 95.486111

CV Set Accuracy: 90.000000

Test Set Accuracy: 97.500000

Iteration 100 | Cost: 8.413299e-01

Training Set Accuracy: 90.625000

CV Set Accuracy: 95.000000

Test Set Accuracy: 97.500000

Iteration 50 | Cost: 8.483802e-01

Training Set Accuracy: 90.277778

CV Set Accuracy: 95.000000

Test Set Accuracy: 97.500000

The trend is that as the Iteration(cost) increases(cost decreases), the training set accuracy increases as expected, but the CV Set/Test Set Accuracy decreases. My initial thought is that this has to do with precision/bias issue, but I really can't buy it.

Anyone know what this entails? Or any reference?

Answer 1

Training scores improving (loss decreasing and accuracy increasing) whilst the opposite happens with cross validation and test data is a sign of overfitting to the training data. Your neural network is getting worse at generalising and no amount of further training will improve it - in fact the situation will get worse the more you train.

This is the main reason you have CV data sets, to show you when this happens before you try to use your model against the test set or real world data. So it is not "peculiar" at all, but the CV set doing its job for you, allowing you notice something has gone wrong.

To improve on the situation, you need to use some form of regularisation. The simplest approach here would be to take your model from around 100 epochs (because it has best CV score that I can see). This is early stopping, and is a simple valid regularisation approach.

Alternatives for neural networks include L2 weight regularisation (also called weight decay) and dropout.

In addition to this, your question states that you only have 300 examples, and more features than examples. This is really tricky to generalise (and hard to tell if you have - for instance it looks like you only have 20 CV examples and 40 test examples, these numbers are very low and prone to giving you inaccurate estimates simply due to chance of what entries are in the data sets). I recommend you look into K-fold cross-validation in order to get more accurate scores, and help you choose the best model for generalisation.