We need to use a loss function for training the neural networks.

In general, the loss function depends only on the desired output $y$ and actual output $\hat{y}$ and is represented as $L(y, \hat{y})$.

As per my current understanding,

Regularization is nothing but using a new loss function $L'(y,\hat{y})$ which must contain a $\lambda$ term (formally called as regularization term) for training a neural network and can be represented as

$$L'(y,\hat{y}) = L(y, \hat{y}) + \lambda \ell(.) $$

where $\ell(.)$ is called regularization function. Based on the definition of function $\ell$ there can be different regularization methods.

Is my current understanding complete? Or is there any other technique in machine learning that is also considered a regularization technique? If yes, where can I read about that regularization?


Regularization is not limited to methods like L1/L2 regularization which are specific versions of what you showed.

Regularization is any technique that would prevent network from overfitting and help network to be more generalizable to unseen data. Some other techniques are Dropout, Early Stopping, Data Augmentation, limiting the capacity of network by reducing number of trainable parameters.

  • 1
    $\begingroup$ L2 regularisation is an interesting case, in that there are two equivalent ways of implementing it. You can use an adjustment to the loss function, or you can decay the weights after each update step $\endgroup$ Jul 27 '21 at 9:41

Also, keep in mind that not just any augmentation of the loss function is a regularization.

For example, you can add terms to a loss function that enforce constraints on the solution but do not prevent overfitting nor facilitate generalization.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.