Should weight decay regularization be divided by the number of samples?

Question

I was watching a video by Andrew Ng about regularization in logistic regression and neural network models.

He uses the following term for regularization to (the sum is over the weights in the network). The regularization term is added to the loss function. Here $m$ is the number of training samples.

$\frac{\lambda}{2m} \sum_{i} \|w_i\|^2$

Why is the term divided by $m$? Looking at other sources I don't see $m$ in the denuminator. Why would we want smaller weights when the batch size is bigger?

I wouldn't have been surprized to see some factor to account for the number of weights but I don't see how dividing by $m$ makes sense.

Answer 1

Let us think about a very simple exemple, polynomial regression. It is not logistic regression nor neural network, but it should help to get the feel of regularization and dataset size.

Imagine we have a typical straight line relationship between $y$ and $x$ with some random noise added. A balanced model would get an intercept, a slope and very low coefficients for higher order powers of the input variable, right?

In our model $y=b+w_1 x+w_2 x^2+...$, $w_i$ should be small for $i>1$.

Now consider extreme cases for the size of the training dataset. If we have a dataset of three points, a parabola will be fitted to those three points with no error. No matter what three points we draw, there will be a parabola exactly fitting that. But with each three point sample from our underlying process, the parabola will change chaotically, sometimes it will point upwards, sometimes downwards, with no relationship to the underlying process, which was a noisy straight line. A strong regularization will be needed to properly estimate a straight line, in order to get an almost zero coefficient for the $x^2$ term.

Let us go now to the other extreme. A huge lot of points from the noisy straight line are taken. What will happen to the $x^2$ coefficient? Well, most likely the effects of the different datapoints can be expected to cancel and there will be a very small concavity/convexity. The need for regularization is not that high.