I have two different implementations with PyTorch of the Atari Pong game using A2C algorithm. Both implementations are similar, but some portion are different.

  1. https://colab.research.google.com/drive/12YQO4r9v7aFSMqE47Vxl_4ku-c4We3B2?usp=sharing

The above code is from the following Github repository: https://github.com/PacktPublishing/Deep-Reinforcement-Learning-Hands-On/blob/master/Chapter10/02_pong_a2c.py It converged perfectly well!

You can find an explanation in Maxim Lapan's book Deep Reinforcement Learning Hands-on page 269

Here is the mean reward curve :

enter image description here

  1. https://colab.research.google.com/drive/1jkZtk_-kR1Mls9WMbX6l_p1bckph8x1c?usp=sharing

The above implementation has been created by me based on the Maxim Lapan's book. However, the code is not converging. There's a small portion of my code that is wrong, but I can't point out what it is. I've been working on that near a week now.

Here is the mean reward curve :

enter image description here

Can someone tell me the problem portion of the code and how can I fix it?


I have decided to test my code with a simpler environment, i.e. Cartpole-v0.

Here is the code : https://colab.research.google.com/drive/1zL2sy628-J4V1a_NSW2W6MpYinYJSyyZ?usp=sharing

Even that code doesn't seem to converge. Still can't see where is my problem.


I think the bug might be in the ExperienceSource class or in the Agent class.


The following question will help you understand the classes ExperienceSource and ExperienceSourceFirstLast.

  • $\begingroup$ Note that programming issues are off-topic here. This issue is in the context of RL, so I may not close it. $\endgroup$
    – nbro
    May 15 '20 at 19:48
  • $\begingroup$ Thanks @nbro! I will keep that in mind. As you allow me, I will leave my question active. Otherwise, I will delete it. $\endgroup$
    – jgauth
    May 15 '20 at 19:52

Here is the commit

I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was doing. It was just repeating the same queue.

In [2]: histories = [deque(maxlen=5)] * 4                                       

In [3]: histories                                                               
Out[3]: [deque([]), deque([]), deque([]), deque([])]

In [4]: histories[0].append(1)                                                  

In [5]: histories                                                               
Out[5]: [deque([1]), deque([1]), deque([1]), deque([1])]

So I just replace it by histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]. That fixed my problem.

In [7]: histories = [deque(maxlen=5) for i in range(4)]                         

In [8]: histories                                                               
Out[8]: [deque([]), deque([]), deque([]), deque([])]

In [9]: histories[0].append(1)                                                  

In [10]: histories                                                              
Out[10]: [deque([1]), deque([]), deque([]), deque([])]

The curve representing the mean reward looks like

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.