Following the online courses with Andrew Ng, he talks about L2 regularization (a.k.a. weight decay) and input normalization. Now, the argument is that L2 regularization make the weights smaller, which makes the sigmoid activation functions (and thus the whole network) "more" linear.

Question 1: can this rather handwavey explanation be formalized? Can we define "more linear" in a mathematical sense, and demonstrate that smaller weights in fact achieve this?

Question 2: in contrast to sigmoid, ReLU activations have a single point where it is nonlinear - i.e. the breaking point at x=0. No scaling of the input changes the shape (i.e. derivative) of this function, the only effect is reducing the magnitude of positive outputs. Does the argument still hold? Why?

Input normalization is given as a good practice, but it seems to me that the network should just compensate for varying magnitude between components of the input by scaling the weights appropriately. The only exception I can think of is again under L2 regularization, which would penalize large weights (assoiciated with small inputs).

Question 3: Is this correct, and is input scaling thus mostly important with L2 normalization, or is there some reason why the network would fail to adjust the weights without scaling?

  • 1
    $\begingroup$ Although all your three questions are related to L2 regularization, I advise you to ask one question per post to facilitate the answerer's life. $\endgroup$ – nbro Jan 22 '20 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.