0
$\begingroup$

I have a confusion on where exactly is the L2 regularization (weight decay) is added.

In various resources I have come across, I find two equations where L2 regularization is applied.

Adding R(W) to loss function makes sense because it tries decrease large weights. Also, I have seen equations where we add R(W) to the weight update term, 2nd equation in 2nd line as shown in this image:

enter image description here

In the above image, using the weight update rule that

W(final) = W(initial) + (alpha) * (Gradient of W),

I obtain a different equation as compared to the other equation which is commonly written in various resources.

Where exactly is the regularization term added, I previously thought it was only added in Loss function but that gives me a different weight update equation from what is commonly presented in resources.( Or is my interpretation of the equation wrong? )

I presume it is also added in weight update equation because while constructing models, we add regularization term.

model.add(Conv2D(256, (5,5), padding="same", kernel_regularizer=l2(reg)))

Would be grateful for any help.

Improve this question
$\endgroup$
1
  • $\begingroup$ As per the keras documentation, "These penalties are summed into the loss function that the network optimizes. " $\endgroup$
    – Hrushi
    Commented Jul 11, 2020 at 12:08

1 Answer 1

Reset to default
1
$\begingroup$

The regularization terms are applied to the loss functions by default. However, their gradients do appear in the update step as the gradient of loss appears in the update step.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for the clarification, but when you say "gradients appear in the update step", shouldn't we have the derivative of R(W) in the weight update equation, but instead we have the R(W) as it is. $\endgroup$
    – Hrushi
    Commented Jul 9, 2020 at 8:19
  • 1
    $\begingroup$ That seems to be the case on the 3rd equation (most bottom). I think there's simply a typo on 2nd equation (upper-right). We should have gradient of R(w) there. $\endgroup$
    – SpiderRico
    Commented Jul 10, 2020 at 3:00
  • $\begingroup$ Yes, I believe it's a mistake from my part, [as per this credible resource ] (neuralnetworksanddeeplearning.com/chap3.html#regularization) $\endgroup$
    – Hrushi
    Commented Jul 11, 2020 at 6:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .