2
$\begingroup$

When using neural networks (NNs), we often normalized the inputs. I think this is done to equally capture the changes in any input feature, that is, if any feature takes huge values and other features take small values, we don't want the NN not to be able to "see" the change in the smaller value.

However, what if we cause the NN to become insensitive to the input, that is, the NN is not able to identify changes in the input because the changes are too small?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ You should also make sure you aren't confusing scaling the inputs with normalizing the inputs. These are very different things. $\endgroup$
    – David Hoelzer
    Aug 13, 2021 at 0:10

1 Answer 1

Reset to default
0
$\begingroup$

When using neural networks (NNs), we often normalized the inputs. I think this is done to equally capture the changes in any input feature, that is, if any feature takes huge values and other features take small values, we don't want the NN not to be able to "see" the change in the smaller value.

However, what if we cause the NN to become insensitive to the input, that is, the NN is not able to identify changes in the input because the changes are too small?

We don't normalize the input to make the model less sensitive to small changes in the input (theoretically, given the correct optimization strategy, the model will learn to approximate the smaller-ranged input as well).

An example of this would be Convolutional Neural Networks. Traditionally, images were represented with integer values ranging from $0$ to $255$. This means that a given pixel could have only $256$ distinct values. However, assuming we normalize the input, let's say to $[0, 1]$, this gives the pixel a whole range of values to occupy, making the input more sensitive to changes.

Instead, normalization is done to help with the model's convergence.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I do not disagree with you, but what you are describing is scaling, not normalizing. Normalizing the input implies that you are forcing it to take more of a Gaussian distribution's shape. Scaling is simply forcing the values to be in a specific range. $\endgroup$
    – David Hoelzer
    Aug 13, 2021 at 0:11
  • $\begingroup$ Hi, in the context of data science normalization means bringing the inputs to take a range of $[0, 1]$ (e.g. see here). Normalization is just a type of feature scaling. $\endgroup$
    – Djib2011
    Aug 13, 2021 at 5:50
  • $\begingroup$ I don't have time to find a better reference for you at the moment, but I'd refer you to towardsai.net/p/data-science/… for now $\endgroup$
    – David Hoelzer
    Aug 13, 2021 at 13:00
  • $\begingroup$ Under different contexts, the term "normalization" can mean different things Most people in data science when saying normalization mean scaling to $[0, 1]$. You could do a google search on "normalization data science", "feature scaling" or something like that to see what I'm talking about. $\endgroup$
    – Djib2011
    Aug 13, 2021 at 14:03
  • $\begingroup$ I totally get what you're saying. It's just not my experience. The distinction is pretty important because when you normalize (gaussian) your training data, it becomes very difficult to apply the same transformation to reality. Scaling, however, is trivial. :) $\endgroup$
    – David Hoelzer
    Aug 13, 2021 at 14:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .