3
$\begingroup$

Status:

For a few weeks now, I have been working on a Double DQN agent for the PongDeterministic-v4 environment, which you can find here.

A single training run lasts for about 7-8 million timesteps (about 7000 episodes) and takes me about 2 days, on Google Collab (K80 Tesla GPU and 13 GB RAM). At first, I thought this was normal because I saw a lot of posts talking about how DQNs take a long time to train for Atari games.

Revelation:

But then after cloning the OpenAI baselines repo, I tried running python -m baselines.run --alg=deepq --env=PongNoFrameskip-v4 and this took about 500 episodes and an hour or 2 to converge to a nice score of +18, without breaking a sweat. Now I'm convinced that I'm doing something terribly wrong but I don't know what exactly.

Investigation:

After going through the DQN baseline code by OpenAI, I was able to note a few differences:

  • I use the PongDeterministic-v4 environment but they use the PongNoFrameskip-v4 environment
  • I thought a larger replay buffer size was important, so I struggled (with the memory optimization) to ensure it was set to 70000 but they set it to a mere 10000, and still got amazing results.
  • I am using a normal Double DQN, but they seem to be using a Dueling Double DQN.

Results/Conclusion

I have my doubts about such a huge increase in performance with just these few changes. So I know there is probably something wrong with my existing implementation. Can someone point me in the right direction?

Any sort of help will be appreciated. Thanks!

$\endgroup$
4
$\begingroup$

Dueling architectures create bigger differences in the values of actions in the state space. This is because the state-value V(s) function is estimated separately from the state-action value Q(s, a). A new quantity, the advantage of an action, can then be defined as A(s, a) = Q(s, a) - V(s).

The Q function, however, measures the the value of choosing a particular action when in this state. The advantage function subtracts the value of the state from the Q function to obtain a relative measure of the importance of each action.

Dueling Network Architectures for Deep Reinforcement Learning


To better direct you, here are 2 resources that could really help you understand why those differences are important.

Speeding up DQN on PyTorch: how to solve Pong in 30 minutes

The main points of the blog are:

  1. Use a larger batch size and play several steps before updating
  2. Play and train in a separate process
  3. Use asyncronous cuda transfers

RL Adventure

This github library has easy to follow jupyter notebooks and links to all of the papers. It includes:

  • DQN
  • Double DQN
  • Dueling DQN
  • Prioritized Experience Replay
  • Noise Networks for Exploration
  • Distributional RL
  • Rainbow (That network that deepmind made that had so many things in it they couldn't find a good name)
  • Distributional RL with Quantile Regression
  • Hierarchical Deep RL
$\endgroup$
  • $\begingroup$ Thanks for your response. Are you telling me that this sort of performance, even anything close to it, is impossible using a normal DDQN? $\endgroup$ – hridayns Jan 30 at 15:40
  • $\begingroup$ Yep. Its not that a DDQN is a bad architecture. Its great and does what it was intended to do (i.e., prevent maximization bias (see section 6.7 in incompleteideas.net/book/bookdraft2017nov5.pdf)). But there exist qualities that an environment can posses which, when taken into consideration, allow a method to perform better within the same amount of computation time as methods which do not consider these qualities. $\endgroup$ – Jaden Travnik Jan 30 at 16:02
  • $\begingroup$ I like this answer so much, because it doesn't even mentioned domain specific knowledge which would speed up the policy search. And the references to external blog post is great, because there are even more complicated mathematical terms available which are promise that Reinforcement learning will run great on graphics cards. $\endgroup$ – Manuel Rodriguez Jan 30 at 16:10
3
$\begingroup$

Although what @Jaden said may be true by itself, it does not really serve to answer my question as I have seen after conducting numerous experiments, and finally reaching close to Dueling Network performance using a normal Double DQN (DDQN).

I made the following changes to my code after closely examining the OpenAI baselines code:

  • Used PongFrameskip-v4 instead of PongDeterministic-v4
  • Used a small replay buffer of size 10000
  • During a step_update() or replay() call, changed the condition for a return from buffer_fill_size < learn_start to t < learn_start, where t is the current timestep, buffer_fill_size is the current size of buffer that has been filled up with experience tuples, and learn_start is the number of timesteps to wait before starting to learn from the experience collected.
  • Made sure that the make_atari() wrapper function is also called on the env:

    ENV_GYM = 'PongFrameskip-v4'
    env = make_atari(ENV_GYM)
    env = wrap_deepmind(env, frame_stack=True, scale=False)
    

    These wrappers may be implemented from scratch or can be obtained from the OpenAI baseline Atari wrappers. I personally used the latter since there is no point in reinventing the wheel.

Conclusion:

The biggest step that I overlooked, or rather didn't pay much attention to was the input preprocessing. These few changes improved my DDQN from an average score saturation at -13 in almost 5000 episodes to +18 in about 700-800 episodes. That is indeed a huge difference. You can check out my implementation here.

$\endgroup$
  • $\begingroup$ Could you mention specifically what was different between your original input processing and the improved version? Was this about scaling of NN inputs for instance? $\endgroup$ – Neil Slater Feb 10 at 10:58
  • 1
    $\begingroup$ By input processing, I mean the environment chosen (PongFrameskip-v4, instead of PongDeterministic-v4 in this case) and the wrapper methods make_atari and wrap_deepmind. I did not have the make_atari wrapper. $\endgroup$ – hridayns Feb 10 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.