# What are the consequences when we multiply, instead of add, a penalty term?

The typical objective function in regression problems like Lasso or Ridge includes a Residual Sum of Squares (RSS) term added to a penalty term based on a norm of the coefficients.

What are the consequences of modifying the objective function by multiplying (instead of adding) the Residual Sum of Squares (RSS) term with a p-norm?

For example, considering the Lasso (L1 norm) and Ridge (L2 norm), we have:

• Lasso (L1 norm): $$\text{Objective} = \text{RSS} \times \left( \lambda \sum |\beta_j| \right)$$
• Ridge (L2 norm): $$\text{Objective} = \text{RSS} \times \left( \lambda \sum \beta_j^2 \right)$$

Would this model change any behavior characteristics of the solution, like optimization process, sparsity or robustness, from the standard Lasso or Ridge? I.e., what are the known implications? Is there a name for this, or practical uses?

• Sorry, just wanted to add a small point on the answer I gave... there exists a similar thing to what you are proposing in the optimization community en.wikipedia.org/wiki/Iteratively_reweighted_least_squares, but the function that you are multiplying must be something non trivial (like regularization).... maybe you are interested in, so wanted to let you know Commented Nov 20, 2023 at 12:11
• @Alberto Interesting, thanks. Commented Nov 20, 2023 at 15:01

Well let's consider one, Ridge regression.
We have 2 terms:

• the regression loss $$L^{pred} = \sum(f(x) - y)^2$$, which we can see that it is a sum of squared values, thus $$L^{pred} \ge 0$$
• the regularization loss $$L^{reg} = \sum w_i^2$$, which we can also see that it's composed by a sum of squared values, thus $$L^{reg} \ge 0$$

Now, we have Ridge, which would sum them $$L = L^{pred} + L^{reg}$$, which will ask the model to "balance" the use of the parameters with the predictions.

However, considering the case where we multiply them, we have $$L = L^{pred} \cdot L^{reg}$$, and knowing that the two terms are greater or equal then zero, we have $$L \ge 0$$.

Now, since we want to minimize the loss, and the loss at most is 0, then having a 0 loss will be a minima of such loss.

How can we make $$L = L^{pred} \cdot L^{reg} = 0$$?
Well, we just need one of the two terms to be zero for a multiplication being zero, and we can notice how $$L^{reg}$$ can be easily be zeroed, just by putting the parameters of your model ($$\beta$$ in your formula) to 0

So, multiplying the loss by a regularization based on norms (at least with $$p\ge1$$) will just drive the model to learn the 0-function, by setting all the weights to 0

• Yes, seems very obvious in retrospect, thanks. (A fix may be to do something like objective = RSS + (RSS*p-norm)) Commented Nov 20, 2023 at 15:02
• @BigMistake yes, most of the things in optimization I would say that look obvious in retrospect, but I knew the answer because it was a thing I also though long time ago, so i already went down the rabbit hole (and wrt the fix, idk, that's equivalent to $(1 + norm)*RSS$ and yes it might work, but idk which property would that have) Commented Nov 20, 2023 at 15:23