# When should we use algorithms like Adam as opposed to SGD?

As far as I know, Stochastic Gradient Descent is an optimization algorithm which belongs to the the category of algorithms where hyper-parameters have to be defined beforehand. They are useful in many cases, but there are some cases that the adaptive learning algorithms (like AdaGrad or Adam) might be preferable.

When are algorithms like Adam and AdaGrad preferred over SGD? What are the cons and pros of adaptive algorithms, like Adam, when we compare them with learning algorithms like SGD?

• By the way, your understanding of the difference between SGD and algorithms like Adam is incorrect. In Adam, you also need to specify certain hyper-parameters beforehand. Read the paper An overview of gradient descent optimization algorithms for a gentle overview of optimization algorithms in ML. – nbro Mar 26 at 20:06

• @DuttaA The accuracy stayed low (like $12 \%$) all the time. – nbro Mar 26 at 14:38