Each training run for DDQN agent takes 2 days, and still ends up with -13 avg score, but OpenAi baseline DQN needs only an hour to converge to +18?

Question

Status:

For a few weeks now, I have been working on a Double DQN agent for the PongDeterministic-v4 environment, which you can find here.

A single training run lasts for about 7-8 million timesteps (about 7000 episodes) and takes me about 2 days, on Google Collab (K80 Tesla GPU and 13 GB RAM). At first, I thought this was normal because I saw a lot of posts talking about how DQNs take a long time to train for Atari games.

Revelation:

But then after cloning the OpenAI baselines repo, I tried running python -m baselines.run --alg=deepq --env=PongNoFrameskip-v4 and this took about 500 episodes and an hour or 2 to converge to a nice score of +18, without breaking a sweat. Now I'm convinced that I'm doing something terribly wrong but I don't know what exactly.

Investigation:

After going through the DQN baseline code by OpenAI, I was able to note a few differences:

• I use the PongDeterministic-v4 environment but they use the PongNoFrameskip-v4 environment
• I thought a larger replay buffer size was important, so I struggled (with the memory optimization) to ensure it was set to 70000 but they set it to a mere 10000, and still got amazing results.
• I am using a normal Double DQN, but they seem to be using a Dueling Double DQN.

Results/Conclusion

I have my doubts about such a huge increase in performance with just these few changes. So I know there is probably something wrong with my existing implementation. Can someone point me in the right direction?

Any sort of help will be appreciated. Thanks!

Answer 1

Dueling architectures create bigger differences in the values of actions in the state space. This is because the state-value V(s) function is estimated separately from the state-action value Q(s, a). A new quantity, the advantage of an action, can then be defined as A(s, a) = Q(s, a) - V(s).

The Q function, however, measures the the value of choosing a particular action when in this state. The advantage function subtracts the value of the state from the Q function to obtain a relative measure of the importance of each action.

Dueling Network Architectures for Deep Reinforcement Learning

---

To better direct you, here are 2 resources that could really help you understand why those differences are important.

Speeding up DQN on PyTorch: how to solve Pong in 30 minutes

The main points of the blog are:

1. Use a larger batch size and play several steps before updating
2. Play and train in a separate process
3. Use asyncronous cuda transfers

RL Adventure

This github library has easy to follow jupyter notebooks and links to all of the papers. It includes:

• DQN
• Double DQN
• Dueling DQN
• Prioritized Experience Replay
• Noise Networks for Exploration
• Distributional RL
• Rainbow (That network that deepmind made that had so many things in it they couldn't find a good name)
• Distributional RL with Quantile Regression
• Hierarchical Deep RL

Answer 2

Although what @Jaden said may be true by itself, it does not really serve to answer my question as I have seen after conducting numerous experiments, and finally reaching close to Dueling Network performance using a normal Double DQN (DDQN).

I made the following changes to my code after closely examining the OpenAI baselines code:

• Used PongFrameskip-v4 instead of PongDeterministic-v4
• Used a small replay buffer of size 10000
• During a step_update() or replay() call, changed the condition for a return from buffer_fill_size < learn_start to t < learn_start, where t is the current timestep, buffer_fill_size is the current size of buffer that has been filled up with experience tuples, and learn_start is the number of timesteps to wait before starting to learn from the experience collected.

• Made sure that the make_atari() wrapper function is also called on the env:

  ENV_GYM = 'PongFrameskip-v4'
  env = make_atari(ENV_GYM)
  env = wrap_deepmind(env, frame_stack=True, scale=False)
  

  These wrappers may be implemented from scratch or can be obtained from the OpenAI baseline Atari wrappers. I personally used the latter since there is no point in reinventing the wheel.

Conclusion:

The biggest step that I overlooked, or rather didn't pay much attention to was the input preprocessing. These few changes improved my DDQN from an average score saturation at -13 in almost 5000 episodes to +18 in about 700-800 episodes. That is indeed a huge difference. You can check out my implementation here.