2
$\begingroup$

In contrast to L2 regularization, L1 regularization usually yields sparse feature vectors and most feature weights are zero.

What's the reason for the above statement - could someone explain it mathematically, and/or provide some intuition (maybe geometric)?

$\endgroup$
1
$\begingroup$

In L1 regularization, the penalty term you compute for every parameter is a function of the absolute value of a given weight (times some regularization factor). Thus, irrespective of whether a weight is positive or negative (due to the absolute value) and irrespective of how large the weight is, there will be a penalty incurred as long as weight is unequal 0. So, the only way how a training procedure can considerably reduce the L1 regularization penalty is by driving all (unnecessary) weights towards 0, which results in a sparse representation.

Of course, the L2 regularization will also only be strictly 0 when all weights are 0. However, in L2, the contribution of a weight to the L2 penalty is proportional to the squared value of the weight. Therefore, a weight whose absolute value is smaller than 1, i.e. $abs(weight) < 1$, will be much less punished by L2 than it would be by L1, which means that L2 puts less emphasis on driving all weights towards exactly 0. This is because squaring a some value in (0,1) will result in a value of lower magnitude than taking the un-squared value itself: $x^2 < x\ for\ all\ x\ with\ abs(x) < 1$.

So, while both regularization terms end up being 0 only when weights are 0, the L1 term penalizes small weights with $abs(x) < 1$ much more strongly than L2 does, thereby driving the weight more strongly towards 0 than L2 does.

| improve this answer | |
$\endgroup$
-1
$\begingroup$

When we are talking about sparse vectors, the dimensions of vectors are high and if we go for a feature cross to get non-linearinty for the model then the number of dimensions goes further and most of them we don't need it.

When the model features are high then our model will be easily going to have high variance if we trained too much. We need to do regularization to make the model simple since Most of the time, Simplicity wins

The common regularizations are

  1. L2 regularization where we aimed to reduce the sum of the squares of each weight of the feature Sum of Square of weights

  2. L1 regularization where we aimed to reduce the sum of the Absolute of each weight of the feature Sum of the absolute value of weights

When we use L2 regularization, The weights nearly goes to zero and not exactly zero since the derivate of the function is 2*weights. In the other hand, L1 regularization can make weights zero since the derivate of L1 is a constant. Imagine like always reduce constant from the L1 regularization value which was got from preview iteration and eventually we will end up with zero weights.

For the sparse feature space, we need to reduce the unwanted features so we need to make their weights as zero. This can happen when we use L1 regularization.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.