# Why isn't my implementation of A2C for the the atari pong game converging?

I have two different implementations with PyTorch of the Atari Pong game using A2C algorithm. Both implementations are similar, but some portion are different.

The above code is from the following Github repository: https://github.com/PacktPublishing/Deep-Reinforcement-Learning-Hands-On/blob/master/Chapter10/02_pong_a2c.py It converged perfectly well!

You can find an explanation in Maxim Lapan's book Deep Reinforcement Learning Hands-on page 269

Here is the mean reward curve :

The above implementation has been created by me based on the Maxim Lapan's book. However, the code is not converging. There's a small portion of my code that is wrong, but I can't point out what it is. I've been working on that near a week now.

Here is the mean reward curve :

Can someone tell me the problem portion of the code and how can I fix it?

UPDATE 1

I have decided to test my code with a simpler environment, i.e. Cartpole-v0.

Even that code doesn't seem to converge. Still can't see where is my problem.

UPDATE 2

I think the bug might be in the ExperienceSource class or in the Agent class.

UPDATE 3

The following question will help you understand the classes ExperienceSource and ExperienceSourceFirstLast.

• Note that programming issues are off-topic here. This issue is in the context of RL, so I may not close it. – nbro May 15 at 19:48
• Thanks @nbro! I will keep that in mind. As you allow me, I will leave my question active. Otherwise, I will delete it. – jgauth May 15 at 19:52

Here is the commit

I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was doing. It was just repeating the same queue.

In [2]: histories = [deque(maxlen=5)] * 4

In [3]: histories
Out[3]: [deque([]), deque([]), deque([]), deque([])]

In [4]: histories[0].append(1)

In [5]: histories
Out[5]: [deque([1]), deque([1]), deque([1]), deque([1])]


So I just replace it by histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]. That fixed my problem.

In [7]: histories = [deque(maxlen=5) for i in range(4)]

In [8]: histories
Out[8]: [deque([]), deque([]), deque([]), deque([])]

In [9]: histories[0].append(1)

In [10]: histories
Out[10]: [deque([1]), deque([]), deque([]), deque([])]


The curve representing the mean reward looks like