# Simulating successful trajectories in Montezuma's Revenge turns out to be unsuccessful

I have written code in OpenAI's gym to simulate a random playing in Montezuma's Revenge where the agent randomly samples actions from the action space and tries to play the game. A success for me is defined as the case when the agent is atleast able to successfully retrieve the key (Gets a reward of 100). And such cases I dump in a pickle file. I got 44 successful cases when I kept it to run for a day or so. Here is the code I use to generate the training set :

import numpy
import gym
import cma
import random
import time
import pickle as pkl

env   = gym.make('MontezumaRevenge-ram-v0')

observation = env.reset()

#print(observation)
#print(env.action_space.sample())

obs_dict    = []
action_dict = []
success_ctr = 0

for i in range(0, 1000000):
print('Reward for episode',i+1)
done = False
rew = 0
action_list = []
obs_list    = []
while not done:
action = env.action_space.sample()
observation, reward, done, _ = env.step(action)
action_list.append(action)
obs_list.append(observation)

rew += reward
env.render()
time.sleep(0.01)
if done:
env.reset()
if rew > 0:
success_ctr += 1
print(action_list)
action_dict.append(action_list)
obs_dict.append(obs_list)
pkl.dump(obs_dict, open("obslist.pkl", "wb"))
pkl.dump(action_dict, open("action.pkl", "wb"))

print(rew)
time.sleep(1)

try:
print(obs_dict.shape)
except:
pass

print("Took key:", success_ctr)


I loaded the successful cases from my generated pickle file, and simulated the agent's playing using those exact same cases. But, the agent never receives a reward of 100. I dont understand why this is happening. A little search online suggested it could be because of noise in the game. So, I gave a sleep time, before running each episode. Still, doesn't work. Can someone please explain why is this happening? And suggest a way I could go about generating the training set?

What's exactly the point of time.sleep() in this code? I don't really understand it, you're simply stopping the execution of the program for $$0.01$$ seconds, how will that affect the simulator in any way ? It's not running in parallel, it does one step of the simulation when you call env.step function and returns the next state and reward. Calling sleep function only slows down the program here.

The reason for the failure of successful trajectories, when repeated, is probably because the environment isn't stationary. That means that there are enemies or obstacles moving in the environment. If you simply repeat the trajectory that was successful once the enemies and obstacles might be in different positions and the agent will die. The reason why it succeeded the first time is because the agent got lucky. These are still valid learning trajectories because they were successful. The agent should learn reasons (features) why those trajectories were successful but not learn the trajectories themselves because, as you saw, they don't generalize well. If you plan on doing some kind of supervised learning approach you should also generate variety of unsuccessful trajectories so that the agent can learn what are correct and what are not correct actions depending on the current state in the environment.

• To introduce a deterministic element between two successive runs of the env.step(action) to alleviate the noise in the environment if possible. Jan 27 '20 at 12:54
• sleep function will not affect the randomness of the simulator, it's determined by it's own random number generator. See if you can set the seed of the simulator for repeated deterministic episodes. Jan 27 '20 at 13:21

Since the environment has some randomness in it, purely memorizing a trajectory to victory will not work. You will have to memorize every single trajectory for that to work, and there are an infinite number of them.

So, you will need to add some sort of bias to your learning model - i.e., what to do when the observations in your pickle file don't match the current observation.

Your current setup lends itself well to a case-based reasoning (CBR) approach. The idea of CBR is that you have a memory bank of observation-action pairs and when you see a new observation you look up the memory bank and see if the current observation has been seen before. If so, do that action. The interesting part is when there are no observations that match directly, but there are some that are similar. In this case you choose the most similar. The similarity can be calculated in any number of ways, and it is dependant on the data types. This paper is will provide a good start: https://alumni.media.mit.edu/~jorkin/generals/papers/Kolodner_case_based_reasoning.pdf

• Do you know of any examples of CBR applied to selecting actions based on video frames from a game? Without knowing more, I would be concerned that it will not scale well to that environment. Potentially the OP could reduce the dimensionality of the observations to help apply CBR Jan 28 '20 at 8:37
• You have a point in that the similarity metric used to compare two images would have to be well thought out. But I imagine it can still be done. The other option like you said would be to potentially apply convolutions to the images to extract relevant features. Or do other image processing as a feature extraction technique. Jan 28 '20 at 14:46