Does L1/L2 Regularization help reach an optimum result faster?

Question

I understand that L1 and L2 regularization helps to prevent overfitting. My question is then, does that mean they also help a neural network learn faster as a result?

The way I'm thinking is that since the regularization techniques reduce weights (to 0 or close to 0 depending on whether it's L1 or L2) that are not important to the neural network, this would, in turn, result in "better values" for the output neurons right? Or perhaps I am completely wrong.

For example, suppose I have a neural network that is to train a snake to move around a NxN environment. With regularization, the snake will learn faster in terms of survive longer in the game?

Answer 1

I am not aware of any empirical results regarding this question. But in theory, adding a regularization term shall make the learning task actually even harder, since there is suddenly a second loss term that the network has to be optimized for, which is not even directly related to achieving the original task of fitting the model to the data. It is true that the regularization term will try drive as many of the weights towards low values as it can. But, at the same time, the other loss term (computed on the original optimization criterion) will try to drive many of the weights to larger values unequal 0 in order to achieve the original training task. (If that wasn't the case, you would be good to go without regularization in the first place.) Having these competing interests shall then make achieving the original task harder, which I would in theory expect to be more time consuming than, alternatively, allowing the model overfit until a certain admissible error is reached.

After all, the goal of regularization is to improve generalization of a learned model (i.e. to prevent overfitting, as you said), which is about the quality of the outcome and not about how quickly you get to the desired outcome.