As far as I know, Stochastic Gradient Descent is an optimization algorithm which belongs to the the category of algorithms where hyper-parameters have to be defined beforehand. They are useful in many cases, but there are some cases that the adaptive learning algorithms (like AdaGrad or Adam) might be preferable.

When are algorithms like Adam and AdaGrad preferred over SGD? What are the cons and pros of adaptive algorithms, like Adam, when we compare them with learning algorithms like SGD?

  • 1
    $\begingroup$ By the way, your understanding of the difference between SGD and algorithms like Adam is incorrect. In Adam, you also need to specify certain hyper-parameters beforehand. Read the paper An overview of gradient descent optimization algorithms for a gentle overview of optimization algorithms in ML. $\endgroup$ – nbro Mar 26 '19 at 20:06

Empirically, I observed that algorithms like Adam and RMSProp tended to give me a final higher performance (in my case, the accuracy) on (the validation dataset) with respect to SGD. However, I also observed that Adam and RMSProp are highly sensitive to certain values of the learning rate (and, sometimes, other hyper-parameters like the batch size) and they can catastrophically fail to converge if e.g. the learning rate is too high. On the other hand, in general, SGD have not led me to the highest performance, but they did not catastrophically fail (at least, as much as Adam and RMSProp) in my experiments (even when using quite different hyper-parameters). I noticed that the learning rate (and the batch size) are the hyper-parameters that mainly affect the performance of all these algorithms.

In my experiments, I use SGD without momentum and I used the (PyTorch) default values of Adam and RMSProp. I only compared SGD with Adam and RMSProp, on the relatively simple task of recognising MNIST digits. You can have a look at this repository https://github.com/nbro/comparative-study-between-optimizers, which contains the code I used to perform these experiments. You also have the instructions there to perform the experiments (if you want).

|improve this answer|||||
  • $\begingroup$ I will check out the GitHub repository! Thank you for your answer. $\endgroup$ – Utku Mar 26 '19 at 13:58
  • $\begingroup$ What exactly is 'catastrophically fail to converge'? Do you mean the loss function system becomes unstable and the oscillation amplitude keeps on increasing? $\endgroup$ – DuttaA Mar 26 '19 at 14:32
  • 1
    $\begingroup$ @DuttaA The accuracy stayed low (like $12 \%$) all the time. $\endgroup$ – nbro Mar 26 '19 at 14:38
  • $\begingroup$ Was it multiclass classification? $\endgroup$ – DuttaA Mar 26 '19 at 14:58
  • $\begingroup$ @DuttaA Yeah, it was about recognising 1 of the 10 MNIST digits. $\endgroup$ – nbro Mar 26 '19 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.