We have different kinds of algorithms to optimize the loss like AdaGrad, SGD + Momentum, etc. Some are more commonly used than the others. In some algorithms, they usually range out before they converge, reach to the steepest slope and find the minima. But some of these algorithms are significantly fast. So my question is that the speed is more of a deciding factor here or the route is important too? Or is it just problem dependent?
Here is a picture of what I mean by the Route.