2
$\begingroup$

Why is it that the skewed contour (unscaled features) will result in slow performance of gradient descent? In other words, how (or why) will the gradients end up taking a long time before finding the global minimum in such cases? This might be an obvious question but I'm finding a hard time Visualizing the 3D shapes of the respective contours and relating it to the convergence.

Left one is the contour for the unscaled feature and the right one is scaled (and will apparently converge quickly).

$\endgroup$
1
$\begingroup$

Feature scaling is making your features have an equal representation or say in the final loss of the function. Intuitively as your picture suggests the contour will be elongated along one axes (feature with higher values).

Another example is lets say you have to separate two data clusters, one is (x,2) and another is (x,5) where x is variable. Now if you are using sigmoid activation, sigmoid always gives positive values, but to separate the two classes you need to have negative value for (x,2) in the second last layer, thus the NN will for a longer time have to adjust the weights to be negative (assuming positive weight initialisation) of bias and other features until (x,2) gives negative value in the second last layer. But with feature normalisation the initial value would have been already negative something like (x,-1/3).

EDIT: From my answer on DataScience.SE:

  • Purpose: Normalisation is done so that the Neural Network weights converge faster. In CNN's and Deep Neural Nets this is of particular help especially in CNN this helps to prevent exploding/vanishing gradients.

The most common explanation for normalisation I have come across is that if you have 2 features, one of them has a significant larger scale than the other e.g house price and house area then the feature with a larger scale will dominate the output. This is quite incorrect according to me, since when you back-propagate through the Neural Net the weight updates are directly proportional to the activations, so larger activation means larger feedback and hence weights get reduced faster and become smaller until w1*house price = w2*house area approximately this relation holds true. Yes, it will lead to more oscillations (intuitionally, since learning rate also becomes multiplied with a larger scale) but it will ultimately probably converge.

So the best 3 reasons for using normalisation are:

  • If scale of a feature is large the weights connected to that feature will have larger oscillations resulting in slower convergence and if deep NN is used probably no convergence, whereas normalisation helps in making the values small -1 to 1 so the gradient updates are also small resulting in faster convergence.
  • The best intuition for normalisation can be found from this video of Stanford and its subsequent video. Since we know weight updates is directly proportional to inputs it will also take the sign of the inputs (or exact opposite sign) always. Now we know house price and area is always positive (in our Universe at least!). So the weight updates will always have definite sign either both positive or both negative (depending the sign of downstream gradients). But the weight updates may have a optimal value in the 4th quadrant, so the weight updates will have to follow a zig zag pattern to effectively make a weight update in the 4th quadrant. enter image description here
  • Finally when you are dealing with Deep Neural Nets like CNN if you do not normalise pixels it will result in exponentially large/vanishing gradients. Since generally softmax/sigmoid is used in the last layer, it squashes the outputs. If you have a large output, generally due to un-normalized data, it'll result in wither exact 0 or exact 1 output, which is fed into a log function and BAM! overflow. The error becomes inf or NaN in python. So inf error means exploding gradients and NaN means gradient cannot be calculated. This can probably be remedied by using higher floating point precision but it will result in higher memory and processor consumption ultimately inefficiency.

TL;DR: Normalisation is used for faster weight convergence. Issues faced by un-normalised data are larger weight oscillations, weight updation in non optimal direction, overflow in precision in Deep Neural Nets.

Here are a few great links for better insight:

How and why do normalization and feature scaling work?

Why feature scaling?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.