What are the consequences when we multiply, instead of add, a penalty term?

Question

The typical objective function in regression problems like Lasso or Ridge includes a Residual Sum of Squares (RSS) term added to a penalty term based on a norm of the coefficients.

What are the consequences of modifying the objective function by multiplying (instead of adding) the Residual Sum of Squares (RSS) term with a p-norm?

For example, considering the Lasso (L1 norm) and Ridge (L2 norm), we have:

• Lasso (L1 norm): $$\text{Objective} = \text{RSS} \times \left( \lambda \sum |\beta_j| \right)$$
• Ridge (L2 norm): $$\text{Objective} = \text{RSS} \times \left( \lambda \sum \beta_j^2 \right)$$

Would this model change any behavior characteristics of the solution, like optimization process, sparsity or robustness, from the standard Lasso or Ridge? I.e., what are the known implications? Is there a name for this, or practical uses?

Answer 1

Well let's consider one, Ridge regression.
We have 2 terms:

• the regression loss $L^{pred} = \sum(f(x) - y)^2$, which we can see that it is a sum of squared values, thus $L^{pred} \ge 0$
• the regularization loss $L^{reg} = \sum w_i^2$, which we can also see that it's composed by a sum of squared values, thus $L^{reg} \ge 0$

Now, we have Ridge, which would sum them $L = L^{pred} + L^{reg}$, which will ask the model to "balance" the use of the parameters with the predictions.

However, considering the case where we multiply them, we have $L = L^{pred} \cdot L^{reg}$, and knowing that the two terms are greater or equal then zero, we have $L \ge 0$.

Now, since we want to minimize the loss, and the loss at most is 0, then having a 0 loss will be a minima of such loss.

How can we make $L = L^{pred} \cdot L^{reg} = 0$?
Well, we just need one of the two terms to be zero for a multiplication being zero, and we can notice how $L^{reg}$ can be easily be zeroed, just by putting the parameters of your model ($\beta$ in your formula) to 0

So, multiplying the loss by a regularization based on norms (at least with $p\ge1$) will just drive the model to learn the 0-function, by setting all the weights to 0