1
$\begingroup$

When using neural networks (NNs), we often normalized the inputs. I think this is done to equally capture the changes in any input feature, that is, if any feature takes huge values and other features take small values, we don't want the NN not to be able to "see" the change in the smaller value.

However, what if we cause the NN to become insensitive to the input, that is, the NN is not able to identify changes in the input because the changes are too small?

$\endgroup$
0
$\begingroup$

When using neural networks (NNs), we often normalized the inputs. I think this is done to equally capture the changes in any input feature, that is, if any feature takes huge values and other features take small values, we don't want the NN not to be able to "see" the change in the smaller value.

However, what if we cause the NN to become insensitive to the input, that is, the NN is not able to identify changes in the input because the changes are too small?

We don't normalize the input to make the model less sensitive to small changes in the input (theoretically, given the correct optimization strategy, the model will learn to approximate the smaller-ranged input as well).

An example of this would be Convolutional Neural Networks. Traditionally, images were represented with integer values ranging from $0$ to $255$. This means that a given pixel could have only $256$ distinct values. However, assuming we normalize the input, let's say to $[0, 1]$, this gives the pixel a whole range of values to occupy, making the input more sensitive to changes.

Instead, normalization is done to help with the model's convergence.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.