Why isn't my implementation of A2C for the the atari pong game converging?

Question

I have two different implementations with PyTorch of the Atari Pong game using A2C algorithm. Both implementations are similar, but some portion are different.

1. https://colab.research.google.com/drive/12YQO4r9v7aFSMqE47Vxl_4ku-c4We3B2?usp=sharing

The above code is from the following Github repository: https://github.com/PacktPublishing/Deep-Reinforcement-Learning-Hands-On/blob/master/Chapter10/02_pong_a2c.py It converged perfectly well!

You can find an explanation in Maxim Lapan's book Deep Reinforcement Learning Hands-on page 269

Here is the mean reward curve :

2. https://colab.research.google.com/drive/1jkZtk_-kR1Mls9WMbX6l_p1bckph8x1c?usp=sharing

The above implementation has been created by me based on the Maxim Lapan's book. However, the code is not converging. There's a small portion of my code that is wrong, but I can't point out what it is. I've been working on that near a week now.

Here is the mean reward curve :

Can someone tell me the problem portion of the code and how can I fix it?

UPDATE 1

I have decided to test my code with a simpler environment, i.e. Cartpole-v0.

Here is the code : https://colab.research.google.com/drive/1zL2sy628-J4V1a_NSW2W6MpYinYJSyyZ?usp=sharing

Even that code doesn't seem to converge. Still can't see where is my problem.

UPDATE 2

I think the bug might be in the ExperienceSource class or in the Agent class.

UPDATE 3

The following question will help you understand the classes ExperienceSource and ExperienceSourceFirstLast.

Answer 1

Here is the commit

I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was doing. It was just repeating the same queue.

In [2]: histories = [deque(maxlen=5)] * 4                                       

In [3]: histories                                                               
Out[3]: [deque([]), deque([]), deque([]), deque([])]

In [4]: histories[0].append(1)                                                  

In [5]: histories                                                               
Out[5]: [deque([1]), deque([1]), deque([1]), deque([1])]

So I just replace it by histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]. That fixed my problem.

In [7]: histories = [deque(maxlen=5) for i in range(4)]                         

In [8]: histories                                                               
Out[8]: [deque([]), deque([]), deque([]), deque([])]

In [9]: histories[0].append(1)                                                  

In [10]: histories                                                              
Out[10]: [deque([1]), deque([]), deque([]), deque([])]

The curve representing the mean reward looks like