Why did the L1/L2 regularization technique not improve my accuracy?

Question

I am training a multilayer neural network with 146 samples (97 for the training set, 20 for the validation set, and 29 for the testing set). I am using:

• automatic differentiation,
• SGD method,
• fixed learning rate + momentum term,
• logistic function,
• quadratic cost function,
• L1 and L2 regularization technique,
• adding some artificial noise 3%.

When I used the L1 or L2 regularization technique, my problem (overfitting problem) got worst.

I tried different values for lambdas (the penalty parameter 0.0001, 0.001, 0.01, 0.1, 1.0 and 5.0). After 0.1, I just killed my ANN. The best result that I took was using 0.001 (but it is worst comparing the one that I didn't use the regularization technique).

The graph represents the error functions for different penalty parameters and also a case without using L1.

and the accuracy

What can be?

Answer 1

You have a small dataset. Should you even be using neural nets? Have you done any diagnostics to see if you even have enough data? Are you using the right metric? Accuracy is not always the correct metric. Which weights are you retaining? You will overfit if you save the weights that produce the lowest training error. Save the weights that produced the lowest validation error. L1, L2, and dropout are all great. So many things not described in the problem...

http://www.ultravioletanalytics.com/blog/kaggle-titanic-competition-part-ix-bias-variance-and-learning-curves

I'm wondering why you're not trying interpretable models to see if the resulting weights for the features make sense. Also if your comparing all those models and parameters, set your random initial starting point to be the same by setting the seed. I also hope you are using the same training set for each model.

You probably need more data...

Answer 2

Your network without regularization does not appear to be over fitting but rather it appears to be converging to a minima. I am actually a bit surprised it is doing as well as it is given that your data set is small. So You don't need regularization. If you want to improve the accuracy you might try using an adjustable learning rate. The Keras call back ReduceLROnPlateau can be used for this. Documentation is here. Also use the callback ModelCheckpoint to save the model with the lowest validation loss. Documentation is here. It would help a lot if you posted your model code. I have found if you do encounter over fitting dropout works more effectively than regularization.