3

In L1 regularization, the penalty term you compute for every parameter is a function of the absolute value of a given weight (times some regularization factor). Thus, irrespective of whether a weight is positive or negative (due to the absolute value) and irrespective of how large the weight is, there will be a penalty incurred as long as weight is unequal 0....


3

The following graph shows the constraint region (green), along with contours for Residual sum of squares (red ellipse). These are iso-lines signifying that points on an ellipse have the same RSS. Figure: Lasso (left) and Ridge (right) Constraints [Source: Elements of Statistical Learning] As Ridge regression has a circular constraint ($\beta_1^2 + \beta_2^...


1

The answer is largely the same whether we consider $\ell_1$ or $\ell_2$ regularisation, so I will just speak generally about regularisation. Mean square error for training data Given some training data $\{(x_i, y_i)\}_{i = 1}^n$, a linear regression line $Y = aX + b$ fit using the least squares method looks for coefficients that minimise the sum of squares, ...


Only top voted, non community-wiki answers of a minimum length are eligible