As I understand, this is the general summary of the Regularization-Overfitting Problem:
The classical "Bias-Variance Tradeoff" suggests that complicated models (i.e. models with more parameters, e.g. neural networks with many layers/weights) are able to well capture complicated patterns in data (i.e. low bias) but are unable to generalize well to unseen data (i.e. high variance). On the other hand, simpler models are able to generalize better to unseen data (i.e. low variance), but unable to capture complex patterns in data (i.e. high bias).
Regularization tries to navigate this compromise by attempting to improve the ability of complicated models to generalize to unseen data. Regularization does this by making "complex models simpler", by strategically reducing the number of parameters in complex models such that they maintain their ability to capture complexity in the data but also generalize to unseen data.
Regularization does this by bringing some of the model parameters towards 0 (L1 Regularization) or by bringing many of the model parameters somewhat towards 0 (L2 Regularization). This "shrinkage" effectively negates the influence of some of the parameters in complex models - and as a result, regularized models tend to have "sparser" solutions (i.e. contain more model parameters with values closer to 0).
Regarding this, I am still not sure if the mathematics behind why sparser models might result in less overfitting is clearly known.
The way I currently see things, Regularization seems to be more of a general heuristic : Countless evidence shows that models overfit less when you add a "regularization penalty term" to the model's Loss Function - and thus deliberately choose model parameters corresponding to a region of the Loss Function that is situated away from the true minimum point. Mathematically, I can understand how this happens.
But are there any mathematical justifications that suggest a sparser model based on a regularized solution is less likely to overfit data compared to a non-regularized solution - or is this still based on heuristics and anecdotal evidence? Do we have any insights as to how the Mathematics of Regularization acts to prevent Overfitting?