# DQN Breakout adding an extra negative reward to help training?

I'm trying to train a DQN, so I'm using OpenAI gym and Breakout (Breakout-v0).

I have altered the reward supplied by the environment: If the episode is not completed fully, the agent gets a -10 reward. Could be this counterproductive for learning?

• Are you adding the -10 reward yourself, or is it part of the environment as presented by OpenAI gym? – Neil Slater Nov 21 '18 at 14:31
• I'm adding it when the episode is finished and its not done – JCP Nov 21 '18 at 17:59

If the aim here is to try and get the agent to speed up and finish an episode, a simpler trick might be to reduce the discount factor (e.g. from $$0.999$$ to $$0.99$$). This will cause the agent to focus on getting more short-term rewards, at the expense of long-term planning. In some environments this could be a problem of a different kind, but when the rewards are not sparse and there are not any special high-reward states that need extended setup, it should be OK.
If the aim is to punish losing a "game life" more severely, then this is less likely to cause a problem (because the state will clearly show what to expect), although it may change what the optimal behaviour is, or how it is approached. In general the impact would be reduced risk-taking, and even though Breakout is a deterministic game, the agent is presented with a stochastic environment because each action lasts a random number of frames (2-4). With a strong penalty for losing a life, I think that the agent will be less likely to try to get high bounce angles by hitting with extreme edge of the bat. Note that you don't need to punish the agent for losing a life, it learns that because the episode ends and it cannot score any more points. Value functions are all about predicting future rewards, so the value of any life-losing state is always $$0$$ anyway.