0
$\begingroup$

I have the following question on "Regularization vs. Constrained Optimization" :

In the context of statistical modelling, we are often taught about "Regularization" as a method of dealing with the "Bias-Variance Tradeoff" (i.e. stabilizing the inconsistent performance of complicated models). When a L1-Norm or L2-Norm Penalty Term is added to the estimation function (corresponding to the statistical model) being optimized, some of the model parameters will either "shrink" in size towards 0, thus producing a "sparser" model that is more likely to retain its "low bias" but possible reduce its "high variance":

enter image description here

I have often heard of functions containing these L1-Norm and L2-Norm "Penalty Terms" being referred to as "optimization constraints" (i.e. the "feasible region" from which valid choices of model parameters can belong to has now been "altered" due to these "norm penalty constraints"):

enter image description here

My Question: When we estimate some statistical model's parameters and the estimation equation contains some "regularization penalty term," would it be incorrect to refer to this as an example of "constrained optimization"?

Is regularized optimization in Machine Learning and Statistical Modelling fundamentally any different (with the exception of usually being more difficult and solved using approximate stochastic iterative methods) from Constrained Optimization in Linear Programming?

enter image description here

References:

Improve this question
$\endgroup$
1
  • $\begingroup$ Yes, it can be considered as a type of 'soft' constraint. For example, with these norm regularizations, you bias the learning process towards regions where parameter values are smaller. $\endgroup$
    – SpiderRico
    Commented Feb 14, 2022 at 23:52

1 Answer 1

Reset to default
1
$\begingroup$

Regularization is different than constraints. The problem with an L1 regularization term would be \begin{align*} \underset{\theta}{\min} & \quad f(\theta) + c\lVert \theta \rVert_1 \end{align*} whereas the same problem with an L1 constraint would be \begin{align*} \underset{\theta}{\min} & \quad f(\theta) \\ \text{such that } & \quad \lVert \theta \rVert_1 \leq M \\ & \quad \end{align*} where $c$ and $M$ are constants which needs to be specified. These are different problems, with different values of $\theta$ for solutions, and need to be solved differently (in particular, the constrained problem is significantly harder to solve).

Now, these two problems are related, and we would expect them to have similar solutions, provided $c$ and $M$ are chosen appropriately. You might want to also read this answer on stats exchange

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .