I have a confusion on where exactly is the L2 regularization (weight decay) is added.

In various resources I have come across, I find two equations where L2 regularization is applied.

Adding R(W) to loss function makes sense because it tries decrease large weights. Also, I have seen equations where we add R(W) to the weight update term, 2nd equation in 2nd line as shown in this image:

enter image description here

In the above image, using the weight update rule that

W(final) = W(initial) + (alpha) * (Gradient of W),

I obtain a different equation as compared to the other equation which is commonly written in various resources.

Where exactly is the regularization term added, I previously thought it was only added in Loss function but that gives me a different weight update equation from what is commonly presented in resources.( Or is my interpretation of the equation wrong? )

I presume it is also added in weight update equation because while constructing models, we add regularization term.

model.add(Conv2D(256, (5,5), padding="same", kernel_regularizer=l2(reg))) 

Would be grateful for any help.

  • $\begingroup$ As per the keras documentation, "These penalties are summed into the loss function that the network optimizes. " $\endgroup$ – Hrushi Jul 11 '20 at 12:08

The regularization terms are applied to the loss functions by default. However, their gradients do appear in the update step as the gradient of loss appears in the update step.

  • $\begingroup$ Thank you for the clarification, but when you say "gradients appear in the update step", shouldn't we have the derivative of R(W) in the weight update equation, but instead we have the R(W) as it is. $\endgroup$ – Hrushi Jul 9 '20 at 8:19
  • 1
    $\begingroup$ That seems to be the case on the 3rd equation (most bottom). I think there's simply a typo on 2nd equation (upper-right). We should have gradient of R(w) there. $\endgroup$ – SpiderRico Jul 10 '20 at 3:00
  • $\begingroup$ Yes, I believe it's a mistake from my part, [as per this credible resource ] (neuralnetworksanddeeplearning.com/chap3.html#regularization) $\endgroup$ – Hrushi Jul 11 '20 at 6:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.