The most commonly-used optimizer in Deep Reinforcement Learning research the past few years is probably ADAM (or its AMSGrad variant, which in most frameworks like keras/tensorflow/pytorch etc. can be used by setting an
amsgrad flag to
True in the construction of an ADAM optimizer, and I believe is often also already set to
True by default). This is a somewhat newer optimizer which isn't included in that visualization you linked to.
RMSProper is also still quite popular, and there may be some arguments that it might be better suited for non-stationary learning problems (which we typically have in RL, our learning targets tend to be non-stationary due to the agent adapting its behaviour over time).
All of these more "fancy" optimizers, typically including some form of momentum term, can in fact be especially useful for optimization in loss "landscapes" that are not smooth. Imagine that your loss landscape looks like a tall mountain, where we'd like our optimizer to "glide" all the way down. Now suppose that the overall trend of a side of the mountain is downwards, but that there are lots of little bumps along the way down, that it's not a smooth mountain. The more fancy optimizers with momentum terms are more likely to "jump" over those bumps than a regular SGD optimizer without momentum terms. This is because, when a little bump is encountered, the local gradient may point the wrong way, but a momentum term "remembers" the overall trend of gliding downwards along the mountain and still points in that same direction for a while.