Tag Info

The following graph shows the constraint region (green), along with contours for Residual sum of squares (red ellipse). These are iso-lines signifying that points on an ellipse have the same RSS. Figure: Lasso (left) and Ridge (right) Constraints [Source: Elements of Statistical Learning] As Ridge regression has a circular constraint ($\beta_1^2 + \beta_2^... 2 Here is my take. The larger the$\lambda, the more the corresponding regularization term for a coefficient will be big, so when minimizing the cost function, the coefficient will be reduced by a bigger factor, you can see this effect in the derivation of the update rule for gradient descent for example: \begin{align*} \theta_j := \theta_j - \alpha\ \left[ \... 1 The answer is largely the same whether we consider\ell_1$or$\ell_2$regularisation, so I will just speak generally about regularisation. Mean square error for training data Given some training data$\{(x_i, y_i)\}_{i = 1}^n$, a linear regression line$Y = aX + b\$ fit using the least squares method looks for coefficients that minimise the sum of squares, ...