0
$\begingroup$

I know we should scale the input and output (assuming regression task) before we feed it to the neural network. Then the gradient descent will give the better minima much faster. But I have subtle confusion whether gradient descent with feature scale and without feature scale gives the same result or just gradient descent is not scale-invariant.

$\endgroup$
  • $\begingroup$ More precisely, what do you mean by "scale-invariant" in this case? Gradient descent/ascent is a general method to optimize a function that is differentiable. We usually talk about functions and their gradients. In the context of neural networks, the functions are the loss/error functions. $\endgroup$ – nbro Jul 31 at 12:37
0
$\begingroup$

What are you hoping to get out of the answer for this question? Feature scaling is a method you CAN(but don't have to) use, so that your algorithm performs faster and reaches better general accuracy. I would say that on a simple regression task, where the feature value ranges do not vary a lot, the output would probably be almost the same, but as soon as you introduce data that has one feature of range 1-10, and other range of 10000-100000, that is where you would notice that you NEED to standardize/normalize your features in order to reach optimal results. That's why it's almost a general rule to just scale your data, so your algorithm can generalize better, and you don't have to worry about your algorithm giving higher importance to a feature with higher values, instead of the one with lower values(just an example).

| improve this answer | |
New contributor
Ivan Petrovic Markovic is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
  • 1
    $\begingroup$ I am hoping to know whether the gradient descent is scale invariant or not $\endgroup$ – Swakshar Deb Jul 31 at 10:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.