I have a data set of 1600 examples. I am using 1280 (80%) for training, 160 (10%) for testing, and 160 (10%) for validation. The training goes one of two ways no matter how I fine-tune the L2 parameter:

1) The validation and training error converge, albeit around 75% error

2) The training error settles to around 0%, but the validation error stays around 75%

I don't think my network is too large either. I have trained networks with two hidden layers, both with the same number of nodes as the input. I also tried dropout layers and that did not seem to help.

Does this just mean that I need to add more training examples? Or how do I know that I have reached the limitations of what I am having the network learn?

  • $\begingroup$ How was the validation data selected, randomly or hand selected. I don;t think L2 has anything to do with the problem. Try running the network without L2 and see what the results are. $\endgroup$ – Gerry P Apr 29 '20 at 4:40
  • $\begingroup$ It was selected randomly. I have also tried without a validation set and the training set error is very low while the testing set error is very high $\endgroup$ – pmac Apr 29 '20 at 13:16
  • $\begingroup$ Seems the issue is more complex. Can't give further help without seeing the code. $\endgroup$ – Gerry P Apr 29 '20 at 16:27
  • $\begingroup$ Acutally I am able to get 100% accuracy on the training set and 33% on the testing set with a very small network. Hope you are familiar with Mathematica. func = "ReLU"; net = NetChain[{LinearLayer[in, "Input" -> 50], ElementwiseLayer[func], LinearLayer[l4, "Input" -> 50], ElementwiseLayer[func], SoftmaxLayer["Input" -> 4]}] $\endgroup$ – pmac Apr 29 '20 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.