Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
When using neural networks (NNs), we often normalized the inputs. I think this is done to equally capture the changes in any input feature, that is, if any feature takes huge values and other features take small values, we don't want the NN not to be able to "see" the change in the smaller value.
However, what if we cause the NN to become insensitive to the input, that is, the NN is not able to identify changes in the input because the changes are too small?
When using neural networks (NNs), we often normalized the inputs. I think this is done to equally capture the changes in any input feature, that is, if any feature takes huge values and other features take small values, we don't want the NN not to be able to "see" the change in the smaller value.
However, what if we cause the NN to become insensitive to the input, that is, the NN is not able to identify changes in the input because the changes are too small?
We don't normalize the input to make the model less sensitive to small changes in the input (theoretically, given the correct optimization strategy, the model will learn to approximate the smaller-ranged input as well).
An example of this would be Convolutional Neural Networks. Traditionally, images were represented with integer values ranging from $0$ to $255$. This means that a given pixel could have only $256$ distinct values. However, assuming we normalize the input, let's say to $[0, 1]$, this gives the pixel a whole range of values to occupy, making the input more sensitive to changes.
Instead, normalization is done to help with the model's convergence.
