I have two different implementations with PyTorch of the Atari Pong game using A2C algorithm. Both implementations are similar, but some portion are different.

  1. https://colab.research.google.com/drive/12YQO4r9v7aFSMqE47Vxl_4ku-c4We3B2?usp=sharing

The above code is from the following Github repository: https://github.com/PacktPublishing/Deep-Reinforcement-Learning-Hands-On/blob/master/Chapter10/02_pong_a2c.py It converged perfectly well!

You can find an explanation in Maxim Lapan's book Deep Reinforcement Learning Hands-on page 269

Here is the mean reward curve :

enter image description here

  1. https://colab.research.google.com/drive/1jkZtk_-kR1Mls9WMbX6l_p1bckph8x1c?usp=sharing

The above implementation has been created by me based on the Maxim Lapan's book. However, the code is not converging. There's a small portion of my code that is wrong, but I can't point out what it is. I've been working on that near a week now.

Here is the mean reward curve :

enter image description here

Can someone tell me the problem portion of the code and how can I fix it?


I have decided to test my code with a simpler environment, i.e. Cartpole-v0.

Here is the code : https://colab.research.google.com/drive/1zL2sy628-J4V1a_NSW2W6MpYinYJSyyZ?usp=sharing

Even that code doesn't seem to converge. Still can't see where is my problem.


I think the bug might be in the ExperienceSource class or in the Agent class.


The following question will help you understand the classes ExperienceSource and ExperienceSourceFirstLast.

  • $\begingroup$ Note that programming issues are off-topic here. This issue is in the context of RL, so I may not close it. $\endgroup$
    – nbro
    May 15, 2020 at 19:48
  • $\begingroup$ Thanks @nbro! I will keep that in mind. As you allow me, I will leave my question active. Otherwise, I will delete it. $\endgroup$
    – jgauth
    May 15, 2020 at 19:52

1 Answer 1


Here is the commit

I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was doing. It was just repeating the same queue.

In [2]: histories = [deque(maxlen=5)] * 4                                       

In [3]: histories                                                               
Out[3]: [deque([]), deque([]), deque([]), deque([])]

In [4]: histories[0].append(1)                                                  

In [5]: histories                                                               
Out[5]: [deque([1]), deque([1]), deque([1]), deque([1])]

So I just replace it by histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]. That fixed my problem.

In [7]: histories = [deque(maxlen=5) for i in range(4)]                         

In [8]: histories                                                               
Out[8]: [deque([]), deque([]), deque([]), deque([])]

In [9]: histories[0].append(1)                                                  

In [10]: histories                                                              
Out[10]: [deque([1]), deque([]), deque([]), deque([])]

The curve representing the mean reward looks like

enter image description here


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .