Why do you not see dropout layers on reinforcement learning examples?

I've been looking at reinforcement learning, and specifically playing around with creating my own environments to use with the OpenAI Gym AI. I am using agents from the stable_baselines project to test with it.

One thing I've noticed in virtually all RL examples is that there never seems to be any dropout layers in any of the networks. Why is this?

I have created an environment that simulates currency prices and a simple agent, using DQN, that attempts to learn when to buy and sell. Training it over almost a million timesteps taken from a specific set of data consisting of one month's worth of 5-minute price data it seems to overfit a lot. If I then evaluate the agents and model against a different month's worth of data is performs abysmally. So sounds like classic overfitting.

But is there a reason why you don't see dropout layers in RL networks? Is there other mechanisms to try and deal with overfitting? Or in many RL examples does it not matter? e.g. there may only be one true way to the ultimate high score in the 'breakout' game, so you might as well learn that exactly, and no need to generalise?

Or is it deemed that the chaotic nature of the environment itself should provide enough different combinations of outcomes that you don't need to have dropout layers?

• Have you tried adding dropout to a RL network to see what happens? I have (for a LunarLander_v2 solver that works well otherwise), and the results were terrible, it goes from solving in 400 episodes, to completely failing to solve it ever (and I tried plenty of variations in other params). Other regularisation is OK, and even beneficial. I don't know why the problems with dropout yet though, was thinking of asking a question here . . . Oct 7, 2018 at 12:42
• Neil, yes, it seem that your experience there echos what is said below by Dennis. As I expected, in the 'solve this puzzle' type scenario you really are looking to overfit as you do want to solve that specific puzzle. And you want to know if you are encountering a blue alien vs a red alien as they might behave differently in your game. Oct 8, 2018 at 19:51
• @MattHamilton I just happened to see a thread of tweets someone made about the "RL researchers are evaluating on the training data" story. I've edited a link to that into my answer. The point is still the same when it comes to answering your question, but I do agree with the chain of tweets I linked to there that the truth is a bit more nuanced. Oct 9, 2018 at 17:34

Dropout essentially introduces a bit more variance. In supervised learning settings, this indeed often helps to reduce overfitting (although I believe there dropout is also already becoming less.. fashionable in recent years than in the few years before that; I'm not 100% sure though, it's not my primary area of expertise).

In Reinforcement Learning, additional variance is not really what we're looking for. There already tends to be a large amount of variance in the learning signals that we get, and this variance already tends to be a major issue for learning stability and/or learning speed. For example:

• Randomness in action selection leads to variance in returns that we observe
• There may be randomness inherent to the environment itself, leading to extra variance in our observations (some environments are nondeterministic)
• Unlike Supervised Learning settings, in Reinforcement Learning we often actually use our own predictions as a part of our loss function / training signal. For example, in temporal-difference learning (like Q-learning / DQN), the target that we update towards looks like $$r + \max_{a'} Q(s', a')$$. In that term, only the $$r$$ is a ground-truth observation (like we would use in supervised learning), and the other term is our own prediction. During a learning process, those latter parts (our own predictions) are changing over time. This is a "moving target'' problem, which can be viewed as additional variance in our learning signals.

Many important parts of Deep RL algorithms (without which our training processes empirically turn out to destabilize and break down) are very much tailored towards reducing that variance. For example, Target Networks in DQN were introduced specifically to reduce the moving target problem. From this point of view, it's not surprising that if we were to add more artificial variance through other means again (such as dropout), that this would hurt performance / destabilize learning.

Is there other mechanisms to try and deal with overfitting? Or in many RL examples does it not matter? e.g. there may only be one true way to the ultimate high score in the 'breakout' game, so you might as well learn that exactly, and no need to generalise?

In the majority of current (Deep) Reinforcement Learning research, overfitting is indeed not viewed as a problem. The vast majority of RL research consists of training in one environment (for example Cartpole, or Breakout, or one particular level in Pacman, or navigating in one specific maze, etc.), and either constantly evaluating performance during that learning process, or evaluating performance after such a learning process in the same environment.

If we were to compare that evaluation methodology to what happens in supervised learning... we are basically evaluating performance on the training set*. In supervised learning, this would be absolutely unacceptable, but in RL it is very much treated as acceptable and more rule than exception. Some say this is simply a problem in current RL research, something that needs to change. It could also be argued that it's not necessarily a problem; if we really are able to train the agent in precisely the same environment that we wish to deploy it in later... well, then what's the problem with it overfitting to that environment?

So, when we're using the evaluation methodology described above, indeed we are overfitting to one specific environment, but overfitting is good rather than bad according to our evaluation criteria. It is clear that this methodology does not lead to agents that can generalize well though; if you consistently train an agent to navigate in one particular maze, it will likely be unable to navigate a different maze after training.

*Note: the truth, in my opinion, is slightly more nuanced than that we are really "evaluating on the training set" in RL. See, for example, this nice thread of tweets: https://twitter.com/nanjiang_cs/status/1049682399980908544

I have created an environment that simulates currency prices and a simple agent, using DQN, that attempts to learn when to buy and sell. Training it over almost a million timesteps taken from a specific set of data consisting of one month's worth of 5-minute price data it seems to overfit a lot. If I then evaluate the agents and model against a different month's worth of data is performs abysmally. So sounds like classic overfitting.

Note that your evaluation methodology described here indeed no longer fits the more "common" evaluation methodology. You have a problem with concept drift, with nonstationarity in the environment. This means overfitting may be a problem for you.

Still, I'm not sure if dropout would help (it's still additional variance which may hurt). First and foremost, you'd want to make sure that there's some way to keep track of the time / month in your inputs, such that you'll at least have a chance of learning a policy that adapts itself over time. If you have a clear, solid boundary between "training phase" and "evaluation phase", and you know that concept drift occurs across that boundary (you know that your environment behaves differently in the training phase from the evaluation phase)... you really don't have much hope of learning a policy only from experience in the training phase that still performs well in the evaluation phase. I suspect you'll have to get rid of that clear, solid boundary. You'll want to keep learning throughout the evaluation phase as well. This enables your learning algorithm to actually collect experience in the changed environment, and adapt to it.

• Dennis, thank you very much for your very detailed reply! A lot of it confirms I think my suspicions: ie that more RL is trying to solve a very specific problem in a specific environment in which there is often one clear 'best' solution to that problem. Oct 10, 2018 at 19:09
• @MattHamilton Note that there is research towards RL for more general environments. If you're interested in that stuff, you'll want to search for combinations of "Transfer learning" and "Reinforcement Learning", or things like Multi-Task RL (Multi-Objective RL may also be interesting, but is probably slightly different). Those kinds of efforts do still tend to be significantly less performant than the traditional approach of training and evaluating on one environment at a time though. Oct 10, 2018 at 19:15