When I try to run Amidar even without RL code, I cannot get the environment to move immediately. It takes about 100 steps before the game actually starts moving. I use the following simple code to display some images and print some actions (I always try to do the same action, namely going up):

env = gym.make('Amidar-v0')

for i in range(1000):
    action = 2 
    next_state, reward, terminated, info = env.step(action) # take a random action
    print(f"Timestep {i}")

When running this code, it takes until about step 85 before the environment starts to move. After that, each step, it moves until the agent is hit by the enemy. Then the environment restarts in the start state, and it takes quite some time before it starts to move again. I have tried doing 'FIRE' as my first action; however, this is not working since it also takes a while before the environment starts moving. Because of this, my buffer is almost always filled with the same images and hence my network isn't learning anything. How do I get this environment to move immediately?

  • $\begingroup$ Hi @Lennart and welcome to AI Stack Exchange! It looks like this question was closed because it is off-topic. If you would like to keep this question open, you will need to edit it to meet the on-topic rules discussed here. Just comment here if you would like help in any way with this. I hope to see you around the site! $\endgroup$ – DeepQZero May 25 at 18:57
  • $\begingroup$ Hi @DeepQZero, thanks for the response. I just read the off-topic rules and it was probably closed because the question is about specific software. No need to keep this question open. $\endgroup$ – Lennart May 26 at 19:24

I reviewed some videos of Amidar on YouTube, and it seems that the game screen is fixed for a few seconds before the start of each level. This is probably to give the human player a chance to rest between levels and maybe even pause the game to take a break, as the gameplay looks very intense.

How do I get this environment to move immediately?

I suggest creating a gym wrapper to change the reset function of the environment to produce a different start state. Since the initial 85 frames are not influenced by the agent's actions, these frames are unhelpful and unnecessary for your agent's training. You could consider calling the step function 85 times in your new implementation of the reset function using a random action; then return the resultant state. When calling your newly implemented reset function, the agent's start state will be frame 85 instead of frame 0. As you noted, frame 85 is immediately followed with many frames of meaningful movement, which will provide higher-quality training data. In contrast, the original implementation of the reset function starts at frame 0, which is followed by 85 unhelpful fixed frames.

Because of this, my buffer is almost always filled with the same images and hence my network isn't learning anything.

ATARI environments are difficult even for some of the more powerful RL algorithms. For example, it is one of the harder environments for DQN to solve (see the Nature paper, pg. 531). Don't worry if your agent isn't learning right away. It could take hours or even days to learn enough to noticeably improve, even with a correct implementation of a powerful algorithm. Good luck, and welcome to AI Stack Exchange!


Not the answer you're looking for? Browse other questions tagged or ask your own question.