2
$\begingroup$

We have different kinds of algorithms to optimize the loss like AdaGrad, SGD + Momentum, etc. Some are more commonly used than the others. In some algorithms, they usually range out before they converge, reach to the steepest slope and find the minima. But some of these algorithms are significantly fast. So my question is that the speed is more of a deciding factor here or the route is important too? Or is it just problem dependent?

enter image description here

Here is a picture of what I mean by the Route.

$\endgroup$
  • $\begingroup$ Well, this is most likely application depended, what the developer expects from it etc. By route, I guess you mean the test/validation accuracy ? $\endgroup$ – SpiderRico May 31 at 5:20
  • 1
    $\begingroup$ @SpiderRico Thank you for your response. I have added a picture to the question. Check that out for clarification about the route. $\endgroup$ – Ali KHalili May 31 at 5:49
  • $\begingroup$ Ah I see, you're talking of particular trajectories followed by algorithms on the loss surface. In this, I guess, what you're really interested is the generalization capacities of the minima found by these algorithms. Obviously, you'd want to pick the one that generalizes well/ $\endgroup$ – SpiderRico May 31 at 5:51
  • $\begingroup$ @SpiderRico Hmm, just to know we are on the same page, So the route can be a deciding factor too but it depends on the problem, right? $\endgroup$ – Ali KHalili May 31 at 6:32
  • 1
    $\begingroup$ The route/trajectory followed by the optimization algorithm basically depends your dataset and the loss function. However, what really matters, for the purpose of accuracy performance, is the final point which the trajectory converges. $\endgroup$ – SpiderRico May 31 at 6:35
1
$\begingroup$

The route/trajectory followed by the optimization algorithm basically depends your dataset and the loss function. However, what really matters, for the purpose of final accuracy performance, is the final point which the trajectory converges to.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.