What does the line of code "self.buffer[-1] = observation" do in this BufferWrapper class for DQN?

Question

So the code is related to using a buffer

class BufferWrapper(gym.ObservationWrapper):
    def __init__(self, env, n_steps, dtype=np.int):
        super(BufferWrapper, self).__init__(env)
        self.dtype = dtype
        old_space = env.observation_space
        self.observation_space = gym.spaces.Box(old_space.low.repeat(n_steps, axis=0),
                                                old_space.high.repeat(n_steps, axis=0), dtype=dtype)

    def reset(self):
        self.buffer = np.zeros_like(self.observation_space.low, dtype=self.dtype)
        return self.observation(self.env.reset())

    def observation(self, observation):
        self.buffer[:-1] = self.buffer[1:]
        self.buffer[-1] = observation
        return self.buffer

It is used to basically do some image processing so that the DQN is fed some transformation of the image. This link provides some higher-level logic behind some operations.

How can I actually understand what's the reason behind the code? Almost all repos have the exact same lines with no explanation (e.g. atari games GitHub repo).

My specific question is what is the purpose of the line self.buffer[-1] = observation?

In my case, my observation is a (7*1) array and I have to return that in an appropriate manner from the observation function.

The book has some mention of this class but I couldn't understand much from it

Answer 1

I don't know if you're confused about this code because you're not very familiar with Python or reinforcement learning (specifically, DQN and experience replay), but that code should be very clear to you if you know Python, but maybe you're not very familiar with DQN.

Let's take a look at the observation method.

def observation(self, observation):
    self.buffer[:-1] = self.buffer[1:]
    self.buffer[-1] = observation
    return self.buffer

• self.buffer[:-1] = self.buffer[1:] essentially drops/forgets the first (which is also the oldest) observation in the buffer.

• self.buffer[-1] = observation adds the new observation (passed as parameter) as the last element of the buffer

You can execute the following code to see what that method does.

buffer = [10, 7, 3]
print(buffer)

buffer[:-1] = buffer[1:]

observation = 5
buffer[-1] = observation

print(buffer) # [7, 3, 5]

If you are not familiar with the experience replay technique, you can take a look at this or this answers.