I am training a multilayer neural network with 146 samples (97 for the training set, 20 for the validation set, and 29 for the testing set). I am using:
- automatic differentiation,
- SGD method,
- fixed learning rate + momentum term,
- logistic function,
- quadratic cost function,
- L1 and L2 regularization technique,
- adding some artificial noise 3%.
When I used the L1 or L2 regularization technique, my problem (overfitting problem) got worst.
I tried different values for lambdas (the penalty parameter 0.0001, 0.001, 0.01, 0.1, 1.0 and 5.0). After 0.1, I just killed my ANN. The best result that I took was using 0.001 (but it is worst comparing the one that I didn't use the regularization technique).
The graph represents the error functions for different penalty parameters and also a case without using L1.
and the accuracy
What can be?