Following the online courses with Andrew Ng, he talks about L2 regularization (a.k.a. weight decay) and input normalization. Now, the argument is that L2 regularization make the weights smaller, which makes the sigmoid activation functions (and thus the whole network) "more" linear.
Question 1: can this rather handwavey explanation be formalized? Can we define "more linear" in a mathematical sense, and demonstrate that smaller weights in fact achieve this?
Question 2: in contrast to sigmoid, ReLU activations have a single point where it is nonlinear - i.e. the breaking point at x=0. No scaling of the input changes the shape (i.e. derivative) of this function, the only effect is reducing the magnitude of positive outputs. Does the argument still hold? Why?
Input normalization is given as a good practice, but it seems to me that the network should just compensate for varying magnitude between components of the input by scaling the weights appropriately. The only exception I can think of is again under L2 regularization, which would penalize large weights (assoiciated with small inputs).
Question 3: Is this correct, and is input scaling thus mostly important with L2 normalization, or is there some reason why the network would fail to adjust the weights without scaling?
