Simulating successful trajectories in Montezuma's Revenge turns out to be unsuccessful

Question

I have written code in OpenAI's gym to simulate a random playing in Montezuma's Revenge where the agent randomly samples actions from the action space and tries to play the game. A success for me is defined as the case when the agent is atleast able to successfully retrieve the key (Gets a reward of 100). And such cases I dump in a pickle file. I got 44 successful cases when I kept it to run for a day or so. Here is the code I use to generate the training set :

import numpy
import gym
import cma
import random
import time
import pickle as pkl

env   = gym.make('MontezumaRevenge-ram-v0')

observation = env.reset()

#print(observation)
#print(env.action_space.sample())

obs_dict    = []
action_dict = []
success_ctr = 0

for i in range(0, 1000000):
    print('Reward for episode',i+1)
    done = False
    rew = 0
    action_list = []
    obs_list    = []
    while not done:
        action = env.action_space.sample()
        observation, reward, done, _ = env.step(action)
        action_list.append(action)
        obs_list.append(observation)

        rew += reward
        env.render()
        time.sleep(0.01)
        if done:
            env.reset()
            if rew > 0:
                success_ctr += 1
                print(action_list)
                action_dict.append(action_list)
                obs_dict.append(obs_list)
                pkl.dump(obs_dict, open("obslist.pkl", "wb"))
                pkl.dump(action_dict, open("action.pkl", "wb"))

    print(rew)
    time.sleep(1)

try:
    print(obs_dict.shape)
except:
    pass

print("Took key:", success_ctr)

I loaded the successful cases from my generated pickle file, and simulated the agent's playing using those exact same cases. But, the agent never receives a reward of 100. I dont understand why this is happening. A little search online suggested it could be because of noise in the game. So, I gave a sleep time, before running each episode. Still, doesn't work. Can someone please explain why is this happening? And suggest a way I could go about generating the training set?

Answer 1

What's exactly the point of time.sleep() in this code? I don't really understand it, you're simply stopping the execution of the program for $0.01$ seconds, how will that affect the simulator in any way ? It's not running in parallel, it does one step of the simulation when you call env.step function and returns the next state and reward. Calling sleep function only slows down the program here.

The reason for the failure of successful trajectories, when repeated, is probably because the environment isn't stationary. That means that there are enemies or obstacles moving in the environment. If you simply repeat the trajectory that was successful once the enemies and obstacles might be in different positions and the agent will die. The reason why it succeeded the first time is because the agent got lucky. These are still valid learning trajectories because they were successful. The agent should learn reasons (features) why those trajectories were successful but not learn the trajectories themselves because, as you saw, they don't generalize well. If you plan on doing some kind of supervised learning approach you should also generate variety of unsuccessful trajectories so that the agent can learn what are correct and what are not correct actions depending on the current state in the environment.

Answer 2

Since the environment has some randomness in it, purely memorizing a trajectory to victory will not work. You will have to memorize every single trajectory for that to work, and there are an infinite number of them.

So, you will need to add some sort of bias to your learning model - i.e., what to do when the observations in your pickle file don't match the current observation.

Your current setup lends itself well to a case-based reasoning (CBR) approach. The idea of CBR is that you have a memory bank of observation-action pairs and when you see a new observation you look up the memory bank and see if the current observation has been seen before. If so, do that action. The interesting part is when there are no observations that match directly, but there are some that are similar. In this case you choose the most similar. The similarity can be calculated in any number of ways, and it is dependant on the data types. This paper is will provide a good start: https://alumni.media.mit.edu/~jorkin/generals/papers/Kolodner_case_based_reasoning.pdf