I'm trying to wrap my mind around the concepts of regularisation and optimisation in neural nets, especially around their differences.

In my current understanding, regularisation is intended to tackle overfitting whereas optimisation is about convergence.

However, even though regularisation adds terms to the loss function, both approaches seem to do most of their things during the update phase, i.e. they work directly on how weights are updated.

If both concepts are focused on updating weights,

  1. what are the conceptual differences, or why aren't both L2 and Adam, for example, called either optimisers or regularisers?

  2. Can/should I use them together?


1 Answer 1


You are correct.

The main conceptual difference is that optimization is about finding the set of parameters/weights that maximizes/minimizes some objective function (which can also include a regularization term), while regularization is about limiting the values that your parameters can take during the optimization/learning/training, so optimization with regularisation (especially, with $L_1$ and $L_2$ regularization) can be thought of as constrained optimization, but, in some cases, such as dropout, it can also be thought of as a way of introducing noise in the training process.

You should use regularisation when your neural network is big (where big, of course, is not well-defined) and you have little data (where little is also not well-defined). Why do you want to use regularisation in this case? Because big neural networks have more capacity, so they can memorize the training data more easily (i.e. they can over-fit the training data). If the training data is not representative of the whole data distribution (that you are trying to capture with your neural network), then your neural network may fail on other data from that data distribution, i.e. it may not generalize. Regularization techniques, such as $L_1$ and $L_2$ penalties and dropout, limit the complexity of the functions that can be represented by your neural network (remember that, for a specific set of weights $\theta$, a neural network represents a specific function $f_\theta$), so they prevent your neural network from learning a complicated function that would just over-fit the training data.

Of course, you can also use regularisation when you have a very large dataset or your neural network is not that big. However, in principle, the case mentioned above is the case where you will likely need some kind of regularisation. So, as a rule of thumb, the bigger your NN is and the smaller your training dataset is, the more likely you will need some kind of regularisation.

The typical way to visualize that your neural network is over-fitting is to look at the evolution of the loss function (a type of objective function, which you want to minimize) for the training dataset and the validation dataset (a dataset that you do not use for training, i.e. updating the weights of the neural network). If the training loss is very small, while the validation loss is bigger (stays constant or even increases), as you train more the neural network, that's a good sign of over-fitting, and it suggests that you may need some kind of regularisation. However, note that, even with regularisation, it is not guaranteed that you will find a good set of weights that is able to generalize to all data (not seen during training), but it's more likely.

There are other forms of regularization, such as the KL divergence in the context of Bayesian neural networks or variational auto-encoders, but these are more advanced topics that you don't need to know now. In any case, the KL divergence has the same role as the other regularisation techniques that I mentioned above, i.e., in some way, it restricts/limits the possible functions that you can learn.

  • $\begingroup$ thank you. From your explanation that optimisation tries to identify the best parameters and regularisation tries to constrain them, is it correct to state that they can always work together as long as optimisation happens first? in other words, i first identify the parameters and them constrain them. Makes any sense? $\endgroup$ Commented Nov 26, 2020 at 20:34
  • 1
    $\begingroup$ @FelipeMartinsMelo You could do that, but you often do both at the same time, i.e. you include a "regularisation term" in your objective function, then you optimize this "regularized objective function". These explanations apply more to $L_1$ and $L_2$ regularisation than to dropout or other ad-hoc techniques. $L_1$ and $L_2$ regularisation are theoretically motivated as a "Bayesian way" of regularizing the weights (e.g. assuming that the weights are sampled from a normal distribution). You can actually derive $L_2$ regularisation from Bayes' rule by putting a Gaussian on the weights. $\endgroup$
    – nbro
    Commented Nov 26, 2020 at 20:37
  • 1
    $\begingroup$ Maybe this answer will also be useful, especially one of the videos by Hinton that I am linking to there. Btw, in the case of dropout, we can think that regularisation is happening before updating the weights, the drop of the units happens before computing the output of the NN. $\endgroup$
    – nbro
    Commented Nov 26, 2020 at 20:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .