# Neural Network Optimizers in Reinforcement Learning non-well behaved environments

Here, the nice gifs explain how different algorithms approach towards the root. Unfortunately, the environment in the gif is way too simple and real cases have much more complex environments. Also, in reinforcement learning, the solutions should change each moment in a difficult enough environment since things are dynamic.

My question is which optimizer is best for reinforcement learning in such dynamically changing environment? Adadelta should not move beyond local minima so do we have to use SGD or Adadelta with an exploration heuristic? Please let me know in detail your thoughts.

The most commonly-used optimizer in Deep Reinforcement Learning research the past few years is probably ADAM (or its AMSGrad variant, which in most frameworks like keras/tensorflow/pytorch etc. can be used by setting an amsgrad flag to True in the construction of an ADAM optimizer, and I believe is often also already set to True by default). This is a somewhat newer optimizer which isn't included in that visualization you linked to.