1
$\begingroup$

I am trying to build intution regarding neural networks and their working and this is a question, I am interested in:

I understand that we normalise inputs. The reason this is done is to capture the changes in any input feature equally. Meaning if any feature takes on huge values and other features take small values, we don't want the network to not be able to "see" the change in the smaller value. But what if we cause the network to become insensitive to the input? That is the network is not able to identfy changes in the input because the changes are too small?

| improve this question | | | | |
$\endgroup$
0
$\begingroup$

1) I understand that we normalise inputs. The reason this is done is to capture the changes in any input feature equally. Meaning if any feature takes on huge values and other features take small values, we don't want the network to not be able to "see" the change in the smaller value. But what if we cause the network to become insensitive to the input? That is the network is not able to identfy changes in the input because the changes are too small?

We don't normalize the input to make the model less sensitive small changes in the input (theoretically given the correct optimization strategy the model will learn to approximate the smaller-ranged input as well).

An example to this would be Convolutional Neural Networks. Traditionally images were represented with integer values ranging from $0$ to $255$. This means that a given pixel could have only $256$ distinct values. However, assuming we normalize the input, let's say to $[0, 1]$, this gives the pixel a whole range of values to occupy, making the input more sensitive to changes.

Instead normalization is done to help with the model's convergence (more on that later).

2) What can we say if the neural network learns a constant function? That is the change in output (assume regression) is in the 4th or more decimal place for a change in input that is in 1st decimal place (after normalisation) ?

If a network leans a constant function we can say it has high bias, however this isn't the fault of normalization but rather other things (e.g. very low learning rate).

3) Is there any initialisation (like Xavier, etc) that initially produces sufficient changes in output with input? I understand that if that is the right way to go, the network should be able to learn it but still, is there an initialisation that might give it a good start to this end?

Initialization strategies, are meant to help with the convergence of the network (i.e. combat vanishing/exploding gradients). These simply assume that the input is normalized. Neural networks are inherently capable of distinguishing small changes in the input, given the proper training conditions (e.g. sufficient model capacity, learning rate and other optimization parameters, convenient initialization of weights).

| improve this answer | | | | |
$\endgroup$
  • 1
    $\begingroup$ Got it. Thanks. $\endgroup$ – pranav Jun 24 '19 at 11:55
  • $\begingroup$ Hey... Can you elaborate a little on the network learning a constant function? Why it might happen and ways to tackle it? $\endgroup$ – pranav Jun 25 '19 at 3:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.