I have a data set of 1600 examples. I am using 1280 (80%) for training, 160 (10%) for testing, and 160 (10%) for validation. The training goes one of two ways no matter how I fine-tune the L2 parameter:
1) The validation and training error converge, albeit around 75% error
2) The training error settles to around 0%, but the validation error stays around 75%
I don't think my network is too large either. I have trained networks with two hidden layers, both with the same number of nodes as the input. I also tried dropout layers and that did not seem to help.
Does this just mean that I need to add more training examples? Or how do I know that I have reached the limitations of what I am having the network learn?