For a few weeks now, I have been working on a Double DQN agent for the PongDeterministic-v4 environment, which you can find here.

A single training run lasts for about 7-8 million timesteps (about 7000 episodes) and takes me about 2 days, on Google Collab (K80 Tesla GPU and 13 GB RAM). At first, I thought this was normal because I saw a lot of posts talking about how DQNs take a long time to train for Atari games.


But then after cloning the OpenAI baselines repo, I tried running python -m baselines.run --alg=deepq --env=PongNoFrameskip-v4 and this took about 500 episodes and an hour or 2 to converge to a nice score of +18, without breaking a sweat. Now I'm convinced that I'm doing something terribly wrong but I don't know what exactly.


After going through the DQN baseline code by OpenAI, I was able to note a few differences:

  • I use the PongDeterministic-v4 environment but they use the PongNoFrameskip-v4 environment
  • I thought a larger replay buffer size was important, so I struggled (with the memory optimization) to ensure it was set to 70000 but they set it to a mere 10000, and still got amazing results.
  • I am using a normal Double DQN, but they seem to be using a Dueling Double DQN.


I have my doubts about such a huge increase in performance with just these few changes. So I know there is probably something wrong with my existing implementation. Can someone point me in the right direction?

Any sort of help will be appreciated. Thanks!


2 Answers 2


Dueling architectures create bigger differences in the values of actions in the state space. This is because the state-value V(s) function is estimated separately from the state-action value Q(s, a). A new quantity, the advantage of an action, can then be defined as A(s, a) = Q(s, a) - V(s).

The Q function, however, measures the the value of choosing a particular action when in this state. The advantage function subtracts the value of the state from the Q function to obtain a relative measure of the importance of each action.

Dueling Network Architectures for Deep Reinforcement Learning

To better direct you, here are 2 resources that could really help you understand why those differences are important.

Speeding up DQN on PyTorch: how to solve Pong in 30 minutes

The main points of the blog are:

  1. Use a larger batch size and play several steps before updating
  2. Play and train in a separate process
  3. Use asyncronous cuda transfers

RL Adventure

This github library has easy to follow jupyter notebooks and links to all of the papers. It includes:

  • DQN
  • Double DQN
  • Dueling DQN
  • Prioritized Experience Replay
  • Noise Networks for Exploration
  • Distributional RL
  • Rainbow (That network that deepmind made that had so many things in it they couldn't find a good name)
  • Distributional RL with Quantile Regression
  • Hierarchical Deep RL
  • $\begingroup$ Thanks for your response. Are you telling me that this sort of performance, even anything close to it, is impossible using a normal DDQN? $\endgroup$
    – hridayns
    Jan 30, 2019 at 15:40
  • $\begingroup$ Yep. Its not that a DDQN is a bad architecture. Its great and does what it was intended to do (i.e., prevent maximization bias (see section 6.7 in incompleteideas.net/book/bookdraft2017nov5.pdf)). But there exist qualities that an environment can posses which, when taken into consideration, allow a method to perform better within the same amount of computation time as methods which do not consider these qualities. $\endgroup$ Jan 30, 2019 at 16:02

Although what @Jaden said may be true by itself, it does not really serve to answer my question as I have seen after conducting numerous experiments, and finally reaching close to Dueling Network performance using a normal Double DQN (DDQN).

I made the following changes to my code after closely examining the OpenAI baselines code:

  • Used PongFrameskip-v4 instead of PongDeterministic-v4
  • Used a small replay buffer of size 10000
  • During a step_update() or replay() call, changed the condition for a return from buffer_fill_size < learn_start to t < learn_start, where t is the current timestep, buffer_fill_size is the current size of buffer that has been filled up with experience tuples, and learn_start is the number of timesteps to wait before starting to learn from the experience collected.
  • Made sure that the make_atari() wrapper function is also called on the env:

    ENV_GYM = 'PongFrameskip-v4'
    env = make_atari(ENV_GYM)
    env = wrap_deepmind(env, frame_stack=True, scale=False)

    These wrappers may be implemented from scratch or can be obtained from the OpenAI baseline Atari wrappers. I personally used the latter since there is no point in reinventing the wheel.


The biggest step that I overlooked, or rather didn't pay much attention to was the input preprocessing. These few changes improved my DDQN from an average score saturation at -13 in almost 5000 episodes to +18 in about 700-800 episodes. That is indeed a huge difference. You can check out my implementation here.

  • $\begingroup$ Could you mention specifically what was different between your original input processing and the improved version? Was this about scaling of NN inputs for instance? $\endgroup$ Feb 10, 2019 at 10:58
  • 1
    $\begingroup$ By input processing, I mean the environment chosen (PongFrameskip-v4, instead of PongDeterministic-v4 in this case) and the wrapper methods make_atari and wrap_deepmind. I did not have the make_atari wrapper. $\endgroup$
    – hridayns
    Feb 10, 2019 at 15:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .