I'm learning the Logistic Regression and L2 Regularization. The cost function looks like below.

$$J(w) = -\displaystyle\sum_{i=1}^{n} (y^{(i)}\log(\phi(z^{(i)})+(1-y^{(i)})\log(1-\phi(z^{(i)})))$$

And the regularization term is added. ($\lambda$ is a regularization strength)

$$J(w) = -\displaystyle\sum_{i=1}^{n} (y^{(i)}\log(\phi(z^{(i)})+(1-y^{(i)})\log(1-\phi(z^{(i)}))) + \frac{\lambda}{2}\| w \|$$

Intuitively, I know that if $\lambda$ becomes bigger, extreme weights are penalized and weights become closer to zero. However, I'm having a hard time to prove this mathematically.

$$\Delta{w} = -\eta\nabla{J(w)}$$ $$\frac{\partial}{\partial{w_j}}J(w) = (-y+\phi(z))x_j + \lambda{w_j}$$ $$\Delta{w} = \eta(\displaystyle\sum_{i=1}^{n}(y^{(i)}-\phi(z^{(i)}))x^{(i)} - \lambda{w_j})$$

This doesn't show the reason why incrementing $\lambda$ makes weight become closer to zero. It is not intuitive.


Nobody seems to know precisely why regularization works, but here is my take:

The larger the lamba, the more the corresponding regularization term for a coefficient will be big, so when minimizing the cost function, the coefficient will be reduced by a bigger factor, you can see this effect in the derivation of the update rule for gradient descent for example: \begin{align*} \theta_j := \theta_j - \alpha\ \left[ \left( \frac{1}{m}\ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)} \right) + \frac{\lambda}{m}\theta_j \right] &\ \ \ \ \ \ \ \ \ \ j \in \lbrace 1,2...n\rbrace\newline & \end{align*}

\begin{align*} \theta_j := \theta_j (1- \alpha \frac{\lambda}{m})\left( \frac{\alpha}{m}\ \sum_{i=1}^m (h_\theta(x^{(i)}) - y^{(i)})x_j^{(i)} \right) &\ \ \ \ \ \ \ \ \ \ j \in \lbrace 1,2...n\rbrace\newline & \end{align*}

From this derivation it is clear that at every update the coefficients get reduced by a factor that is usually a little less than 1 and directly proportional to lamba, so as lambda gets bigger the weights gets reduced more and more, eventually for very big values of lambda we risk to totally underfit the data, since only the regularization term will remain in the cost function and all the weights will go to zero.

This is for linear regression but it is essentially the same logic also for logistic regression.

This is taken from Andrew Ng course on Coursera


A more mathemathical precise (and complex) traction of the problem can be found in the Bloomberg machine learning course material:


PS: in the derivation of the update rule for gradient descent the lamba should be divided by the number of training examples, this is important in choosing the right lamba because we want this relationship to be inversely proportional, otherwise the coefficient won't be decreased.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.