For a few weeks now, I have been working on a Double DQN agent for the
PongDeterministic-v4 environment, which you can find here.
A single training run lasts for about 7-8 million timesteps (about 7000 episodes) and takes me about 2 days, on Google Collab (K80 Tesla GPU and 13 GB RAM). At first, I thought this was normal because I saw a lot of posts talking about how DQNs take a long time to train for Atari games.
But then after cloning the OpenAI baselines repo, I tried running
python -m baselines.run --alg=deepq --env=PongNoFrameskip-v4 and this took about 500 episodes and an hour or 2 to converge to a nice score of +18, without breaking a sweat. Now I'm convinced that I'm doing something terribly wrong but I don't know what exactly.
After going through the DQN baseline code by OpenAI, I was able to note a few differences:
- I use the
PongDeterministic-v4environment but they use the
- I thought a larger replay buffer size was important, so I struggled (with the memory optimization) to ensure it was set to 70000 but they set it to a mere 10000, and still got amazing results.
- I am using a normal Double DQN, but they seem to be using a Dueling Double DQN.
I have my doubts about such a huge increase in performance with just these few changes. So I know there is probably something wrong with my existing implementation. Can someone point me in the right direction?
Any sort of help will be appreciated. Thanks!