AFAIK, momentum is quite useful when training CNNs, and can speed-up the training substantially without any drop in validation accuracy.
I've recently learned that it is not as helpful for RNNs where plain SGD is preferred.
For example, Deep Learning by Goodfellow et. al says: " .. both of these approaches have been largely replaced by simply using SGD (even without momentum) " Sec 10.11, pg.401.
Author talks about LSTMs and "both of these approaches" refer to second order and first order+momentum methods,respectively, according to my understanding.
What causes this discrepancy?