I have a confusion on where exactly is the L2 regularization (weight decay) is added.
In various resources I have come across, I find two equations where L2 regularization is applied.
Adding R(W) to loss function makes sense because it tries decrease large weights. Also, I have seen equations where we add R(W) to the weight update term, 2nd equation in 2nd line as shown in this image:
In the above image, using the weight update rule that
W(final) = W(initial) + (alpha) * (Gradient of W),
I obtain a different equation as compared to the other equation which is commonly written in various resources.
Where exactly is the regularization term added, I previously thought it was only added in Loss function but that gives me a different weight update equation from what is commonly presented in resources.( Or is my interpretation of the equation wrong? )
I presume it is also added in weight update equation because while constructing models, we add regularization term.
model.add(Conv2D(256, (5,5), padding="same", kernel_regularizer=l2(reg)))
Would be grateful for any help.