I am training a Multilayer Neural Nets with 146 samples (97 for training set, 20 for validation set and 29 for testing set). I am using: automatic differentiation, SGD method, fixed learning rate + momentum term, logistic function, quadratic cost function, L1 and L2 regularization technique, adding some artificial noise 3%.

When I used L1 or L2 regularization technique my problem (overfitting problem) got worst. i tried different values for lambdas (the penalty parameter 0.0001, 0.001, 0.01, 0.1, 1.0 and 5.0). After 0.1 i just killed my ANN. The best result that i took it was using 0.001 (but it is worst comparing the one that i didnt use regularization technique). The graph represent the error functions for different penalty parameters and also a case without using L1.

enter image description here

and the accuracy enter image description here

What can be? Thanks!!

  • $\begingroup$ Welcome to ai.se...It has been found out that regularization of the polynomial form doesn't work well for NN..Instead use dropouts...This makes intuitional sense since the complexity of NN is not much dependent on the weights. $\endgroup$
    – user9947
    Oct 26 '18 at 12:52
  • $\begingroup$ Hello. Thanks for your answer . Do you know where I can find the information (paper) abou L1 and polynomial form? Thanks!! $\endgroup$
    – LVoltz
    Oct 26 '18 at 14:11
  • $\begingroup$ By polynomial form I meant |weights|^n. $\endgroup$
    – user9947
    Oct 26 '18 at 14:25
  • $\begingroup$ I understand! sorry for the misunderstand. But I´d like to know more about why L1 doen not fit well to NN. Do you have some references to recommend me? thanks! $\endgroup$
    – LVoltz
    Oct 26 '18 at 14:56
  • $\begingroup$ Not ALL polynomials. Proofs describe conditions for which convergence is guaranteed or improved, but not all that has not yet been proven is inviable. There may be a large class of polynomials that perform better with L1 or L2 that hasn't been investigated or represented well on the web There could also be conditions on the input that can be proven to give advantage to L1 or L2. $\endgroup$
    – FelicityC
    Oct 26 '18 at 15:48

Regularization is one of the important prerequisites for improving the reliability, speed, and accuracy of convergence, but it is not a solution to every problem. Irregularity in data is only one of many root causes for slow or otherwise inadequate learning results, and as the results in the question indicates, it can reduce reliability, speed, or accuracy in some cases.

For those who are new to this topic, these are a few other root causes.

  • Complexity beyond the capacity of the computing machinery to model adequately
  • Insufficient number of examples for training
  • Poor distribution alignment between data sets (training, testing, validation, production)
  • Saturation of back propagation values in floating point
  • Outliers cause by error in example creation or labeling
  • Local optimization minima in the loss function combined with insufficient stochastic injection in SGD
  • Leaning to heavily on artificial network convergence and neglecting using other known algorithms as part of an overall system architecture
  • Using activation functions or hyper-parameters that are not well tuned to the example set or the model

In the case of regularization, L1 and L2 have known properties that have been proven in theorem form to guarantee convergence in fewer examples or epochs, but they rely on very specific models toward which convergence is targetted. L1 and L2 are not always beneficial for polynomial models. Trying to make them so tends to lead to over-fitting.

The resulting trained networks lack the needed generalization to produce reliable behavior upon receiving data outside the training set. Irregularities might not always be caught during test and verification. Deployed systems can exhibit the signs of over-fitting over time because the network learned time-specific features.

Overfitting can be prevented using dropout regularization, another technique first proposed in Dropout: A Simple Way to Prevent Neural Networks from Overfitting by Srivastava, et al. 2014.

Another two things to look at ...

  1. Is the logistic function is the best choice and for which layers? It seems to be down-trending. More people are using ReLU and its softer varieties because of higher performance shown in a number of problem domains and models.
  2. Is the noise injection clean in that it is a good pseudo noise source and is it configured properly with whatever SGD methods and libraries being used.

I am training a Multilayer Neural Nets with 146 samples (97 for training set, 20 for validation set and 29 for testing set). I am using: automatic differentiation, SGD method, fixed learning rate + momentum term, logistic function, quadratic cost function, L1 and L2 regularization technique, adding some artificial noise 3%.

The question indicated the best results for the current training set and model.

The best result was obtained using a lambda of 0.001 without regularization.

That's the starting point for improving further. Since complete automated convergence is not yet something libraries and frameworks provide, the engineer normally has to converge on the combination of techniques and settings that best help the network's convergence.

  • $\begingroup$ I don't understand...What do mean by convergence? Convergence to a solution for only training set? For training and CV set? For any data provided? $\endgroup$
    – user9947
    Oct 26 '18 at 16:00
  • $\begingroup$ Convergence is a concept applied throughout AI disciplines, including deep networks, fuzzy logic, and real time adaptive control. In this answer, the term is used too many times to explain each one in comments. At least a hundred papers related to MLPs, recurrent networks, and other deep learners can help one better understand convergence. Some offer proofs that a proposed algorithm is convergent under fixed conditions. The idea of convergence comes from calculus. Roughly, "A system is convergent if it approaches a predefined ideal as time approaches infinity for all cases in the system." $\endgroup$
    – FelicityC
    Oct 26 '18 at 16:23
  • $\begingroup$ Yes that's what I am asking, convergence for what? A systems convergence most of the times entirely depend on the input data. $\endgroup$
    – user9947
    Oct 26 '18 at 17:32
  • $\begingroup$ Other dependencies: Properties of algorithms representing method of descent and back-propagation, computing resources available, layer widths, activation functions, loss functions, preparation of data, hyper-parameter settings and control, batching. and a dozen other things. $\endgroup$
    – FelicityC
    Oct 26 '18 at 17:47

You have a small dataset. Should you even be using neural nets? Have you done any diagnostics to see if you even have enough data? Are you using the right metric? Accuracy is not always the correct metric. Which weights are you retaining? You will overfit if you save the weights that produce the lowest training error. Save the weights that produced the lowest validation error. L1, L2, and dropout are all great. So many things not described in the problem...


I'm wondering why you're not trying interpretable models to see if the resulting weights for the features make sense. Also if your comparing all those models and parameters, set your random initial starting point to be the same by setting the seed. I also hope you are using the same training set for each model.

You probably need more data...


Your network without regularization does not appear to be over fitting but rather it appears to be converging to a minima. I am actually a bit surprised it is doing as well as it is given that your data set is small. So You don't need regularization. If you want to improve the accuracy you might try using an adjustable learning rate. The Keras call back ReduceLROnPlateau can be used for this. Documentation is here. Also use the callback ModelCheckpoint to save the model with the lowest validation loss. Documentation is here. It would help a lot if you posted your model code. I have found if you do encounter over fitting dropout works more effectively than regularization.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.