DQN Breakout adding an extra negative reward to help training?

Question

I'm trying to train a DQN, so I'm using OpenAI gym and Breakout (Breakout-v0).

I have altered the reward supplied by the environment: If the episode is not completed fully, the agent gets a -10 reward. Could be this counterproductive for learning?

Answer 1

In general, the approach of adding or altering the reward structure of a RL environment, when you are still trying to create an agent that solves the original problem, is called Reward Shaping.

Reward Shaping can be tricky to get right. In your case I think it may be counter-productive. The main issue is that the agent does not know the current time step as part of the state. So from its perspective, a negative reward for "timing out" actually looks like a random chance of getting that same reward in the kind of states that happen later in the game. The effect may be small enough chance that it doesn't really matter (after all it still gets positive rewards for hitting bricks). However, it may appear as high variance on certain game positions, making the agent learn to avoid some of them for no good reason.

If the aim here is to try and get the agent to speed up and finish an episode, a simpler trick might be to reduce the discount factor (e.g. from $0.999$ to $0.99$). This will cause the agent to focus on getting more short-term rewards, at the expense of long-term planning. In some environments this could be a problem of a different kind, but when the rewards are not sparse and there are not any special high-reward states that need extended setup, it should be OK.

If the aim is to punish losing a "game life" more severely, then this is less likely to cause a problem (because the state will clearly show what to expect), although it may change what the optimal behaviour is, or how it is approached. In general the impact would be reduced risk-taking, and even though Breakout is a deterministic game, the agent is presented with a stochastic environment because each action lasts a random number of frames (2-4). With a strong penalty for losing a life, I think that the agent will be less likely to try to get high bounce angles by hitting with extreme edge of the bat. Note that you don't need to punish the agent for losing a life, it learns that because the episode ends and it cannot score any more points. Value functions are all about predicting future rewards, so the value of any life-losing state is always $0$ anyway.

If you have more time, then there are also several extensions to basic DQN that require less sampled experience to learn an environment, or improve the resulting policies. A good source for those ideas is the paper Rainbow: Combining Improvements in Deep Reinforcement Learning.