2
$\begingroup$

https://stackoverflow.com/questions/36162180/gradient-descent-vs-adagrad-vs-momentum-in-tensorflow

Here, the nice gifs explain how different algorithms approach towards the root. Unfortunately, the environment in the gif is way too simple and real cases have much more complex environments. Also, in reinforcement learning, the solutions should change each moment in a difficult enough environment since things are dynamic.

My question is which optimizer is best for reinforcement learning in such dynamically changing environment? Adadelta should not move beyond local minima so do we have to use SGD or Adadelta with an exploration heuristic? Please let me know in detail your thoughts.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The most commonly-used optimizer in Deep Reinforcement Learning research the past few years is probably ADAM (or its AMSGrad variant, which in most frameworks like keras/tensorflow/pytorch etc. can be used by setting an amsgrad flag to True in the construction of an ADAM optimizer, and I believe is often also already set to True by default). This is a somewhat newer optimizer which isn't included in that visualization you linked to.

RMSProper is also still quite popular, and there may be some arguments that it might be better suited for non-stationary learning problems (which we typically have in RL, our learning targets tend to be non-stationary due to the agent adapting its behaviour over time).

All of these more "fancy" optimizers, typically including some form of momentum term, can in fact be especially useful for optimization in loss "landscapes" that are not smooth. Imagine that your loss landscape looks like a tall mountain, where we'd like our optimizer to "glide" all the way down. Now suppose that the overall trend of a side of the mountain is downwards, but that there are lots of little bumps along the way down, that it's not a smooth mountain. The more fancy optimizers with momentum terms are more likely to "jump" over those bumps than a regular SGD optimizer without momentum terms. This is because, when a little bump is encountered, the local gradient may point the wrong way, but a momentum term "remembers" the overall trend of gliding downwards along the mountain and still points in that same direction for a while.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Amsgrad looks good. I'll check it out. But due to search spaces being too large in neural networks, I think imagining local minimas or even saddle points would be incorrect. We will need to go mathematically and identify things. Momentum's problem is that it goes too much into wrong areas which would ruin performance in a dynamic environment. $\endgroup$
    – sagar_acharya
    Dec 1, 2018 at 11:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .