0
$\begingroup$

The typical objective function in regression problems like Lasso or Ridge includes a Residual Sum of Squares (RSS) term added to a penalty term based on a norm of the coefficients.

What are the consequences of modifying the objective function by multiplying (instead of adding) the Residual Sum of Squares (RSS) term with a p-norm?

For example, considering the Lasso (L1 norm) and Ridge (L2 norm), we have:

  • Lasso (L1 norm): $$\text{Objective} = \text{RSS} \times \left( \lambda \sum |\beta_j| \right)$$
  • Ridge (L2 norm): $$\text{Objective} = \text{RSS} \times \left( \lambda \sum \beta_j^2 \right)$$

Would this model change any behavior characteristics of the solution, like optimization process, sparsity or robustness, from the standard Lasso or Ridge? I.e., what are the known implications? Is there a name for this, or practical uses?

$\endgroup$
2
  • $\begingroup$ Sorry, just wanted to add a small point on the answer I gave... there exists a similar thing to what you are proposing in the optimization community en.wikipedia.org/wiki/Iteratively_reweighted_least_squares, but the function that you are multiplying must be something non trivial (like regularization).... maybe you are interested in, so wanted to let you know $\endgroup$
    – Alberto
    Nov 20, 2023 at 12:11
  • $\begingroup$ @Alberto Interesting, thanks. $\endgroup$
    – BigMistake
    Nov 20, 2023 at 15:01

1 Answer 1

2
$\begingroup$

Well let's consider one, Ridge regression.
We have 2 terms:

  • the regression loss $L^{pred} = \sum(f(x) - y)^2$, which we can see that it is a sum of squared values, thus $L^{pred} \ge 0$
  • the regularization loss $L^{reg} = \sum w_i^2$, which we can also see that it's composed by a sum of squared values, thus $L^{reg} \ge 0$

Now, we have Ridge, which would sum them $L = L^{pred} + L^{reg}$, which will ask the model to "balance" the use of the parameters with the predictions.

However, considering the case where we multiply them, we have $L = L^{pred} \cdot L^{reg}$, and knowing that the two terms are greater or equal then zero, we have $L \ge 0$.

Now, since we want to minimize the loss, and the loss at most is 0, then having a 0 loss will be a minima of such loss.

How can we make $L = L^{pred} \cdot L^{reg} = 0$?
Well, we just need one of the two terms to be zero for a multiplication being zero, and we can notice how $L^{reg}$ can be easily be zeroed, just by putting the parameters of your model ($\beta$ in your formula) to 0

So, multiplying the loss by a regularization based on norms (at least with $p\ge1$) will just drive the model to learn the 0-function, by setting all the weights to 0

$\endgroup$
2
  • $\begingroup$ Yes, seems very obvious in retrospect, thanks. (A fix may be to do something like objective = RSS + (RSS*p-norm)) $\endgroup$
    – BigMistake
    Nov 20, 2023 at 15:02
  • 1
    $\begingroup$ @BigMistake yes, most of the things in optimization I would say that look obvious in retrospect, but I knew the answer because it was a thing I also though long time ago, so i already went down the rabbit hole (and wrt the fix, idk, that's equivalent to $(1 + norm)*RSS$ and yes it might work, but idk which property would that have) $\endgroup$
    – Alberto
    Nov 20, 2023 at 15:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .