0
$\begingroup$

I am trying to solve the cartpole environment (GitHub) using DQN agent. I have been building my own DQN agent by following a tutorial by Jon Krohn.

I am able to solve the environment with a maximum reward of 499. I am using the experience replay method, and I have given the number of episodes as 1000 episodes.

In the tutorial it takes very much less time to solve the environment, whereas for me it's taking almost 2 hours to solve it.

The library I am using for building my model is tensorflow (tf.keras).

Improve this question
$\endgroup$
11
  • $\begingroup$ Could you share your code? $\endgroup$
    – hal9000
    Jul 16, 2022 at 11:10
  • $\begingroup$ Hi @hal9000 , i am sharing the github link here. This is the code I used. github.com/the-deep-learners/TensorFlow-LiveLessons/blob/master/… . I just want to know how much time it takes to solve $\endgroup$
    – Rishidhar Kasam
    Jul 16, 2022 at 15:19
  • $\begingroup$ From my own experience it takes around 5 min to get maximum reward on cartpole. 2 hours seems indeed a bit long. $\endgroup$
    – hal9000
    Jul 16, 2022 at 16:17
  • $\begingroup$ Looking at your code I think it might be because you extract the mini batch and process each part separately. (there are nice ways to process the entire batch at once, making it go faster). Also I think I see you update the network that is used in the prediction directly instead of resyncing every m steps MSE(Q(state,action,param),reward+gamma*Q'(next_state,action,frozen_param)), while it might not cause instability for cartpole, you might want to delay resyncing the frozen parameters with the current parameters as that makes convergences more stable. $\endgroup$
    – hal9000
    Jul 16, 2022 at 16:40
  • 1
    $\begingroup$ Hi @hal9000 , the problem has been solved. Thanks for your help. Apparently, it was an issue with the versions of my libraries and it looks like the most recent versions were the problem. I downgraded them and as you said, it is done within a couple of minutes. Thanks again. $\endgroup$
    – Rishidhar Kasam
    Jul 18, 2022 at 11:06

1 Answer 1

Reset to default
0
$\begingroup$

Will put this here as a summary of the solution.
To get to cartpole reward 500 shouldn't take a modern compute too long (few minutes).
If it runs longer than that it can be because of a lot of ways to speed it up (assuming the code works as it should):

  1. The versions of the libraries can matter a lot, downgrading ended up fixing the problem. (as the creator of the question pointed out).
  2. find better hyper-parameters. (For example: since DQN is quite unstable, one can play around with the syncing hyper-parameter to make it more stable as that may improve convergence (stability and/or speed))
  3. use gather, masks and other tricks so you don't have to use very slow python loops.
  4. use the gpu.
  5. getting better hardware. (unless you run on an actual potato hardware shouldn't be a problem)
Improve this answer
$\endgroup$
1
  • $\begingroup$ Suggest add 6) disable rendering the game during training. Makes a huge difference in my experience. You can still switch it on for rendering a few test episodes $\endgroup$
    – Neil Slater
    Aug 13 at 7:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .