Why does L1 regularization yield sparse features?

In contrast to L2 regularization, L1 regularization usually yields sparse feature vectors and most feature weights are zero.

What's the reason for the above statement - could someone explain it mathematically, and/or provide some intuition (maybe geometric)?

• This is a relatively old question, but, for reproducibility, could you please provide a link to the article/paper where you took this excerpt that you're quoting from?
– nbro
Jan 29 at 22:25
• Oops, at this point, I don't remember. I'll take care next time! Jan 30 at 12:17

Of course, the L2 regularization will also only be strictly 0 when all weights are 0. However, in L2, the contribution of a weight to the L2 penalty is proportional to the squared value of the weight. Therefore, a weight whose absolute value is smaller than 1, i.e. $$abs(weight) < 1$$, will be much less punished by L2 than it would be by L1, which means that L2 puts less emphasis on driving all weights towards exactly 0. This is because squaring a some value in (0,1) will result in a value of lower magnitude than taking the un-squared value itself: $$x^2 < x\ for\ all\ x\ with\ abs(x) < 1$$.
So, while both regularization terms end up being 0 only when weights are 0, the L1 term penalizes small weights with $$abs(x) < 1$$ much more strongly than L2 does, thereby driving the weight more strongly towards 0 than L2 does.