# Where is L2-regularization term applied

I have a confusion on where exactly is the L2 regularization (weight decay) is added.

In various resources I have come across, I find two equations where L2 regularization is applied.

Adding R(W) to loss function makes sense because it tries decrease large weights. Also, I have seen equations where we add R(W) to the weight update term, 2nd equation in 2nd line as shown in this image:

In the above image, using the weight update rule that

W(final) = W(initial) + (alpha) * (Gradient of W),


I obtain a different equation as compared to the other equation which is commonly written in various resources.

Where exactly is the regularization term added, I previously thought it was only added in Loss function but that gives me a different weight update equation from what is commonly presented in resources.( Or is my interpretation of the equation wrong? )

I presume it is also added in weight update equation because while constructing models, we add regularization term.

model.add(Conv2D(256, (5,5), padding="same", kernel_regularizer=l2(reg)))


Would be grateful for any help.

• As per the keras documentation, "These penalties are summed into the loss function that the network optimizes. " – Hrushi Jul 11 at 12:08