My main purpose right now is to train an agent using the A2C algorithm to solve the Atari Breakout game. So far I have succeeded to create that code with a single agent and environment. To break the correlation between samples (i.i.d), I need to have an agent interacting with several environments.

class GymEnvVec():

    def __init__(self, env_name, n_envs, seed=0):
        make_env = lambda: gym.make(env_name)
        self.envs = [make_env() for _ in range(n_envs)]
        [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]

    def reset(self):
        return [env.reset() for env in self.envs]

    def step(self, actions):
        return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))

I can use the class GymEnvVec to vectorize my environment.

So I can set my environments with

envs = GymEnvVec(env_name="Breakout-v0", n_envs=50)

I can get my first observations with

observations = envs.reset()

Pick some actions with

actions = agent.choose_actions(observations)

The choose_actions method might look like

def choose_actions(self, states):
        assert isinstance(states, (list, tuple))

        actions = []
        for state in states:
            probabilities  = F.softmax(self.network(state)[0])
            action_probs = T.distributions.Categorical(probabilities)

        return [action.item() for action in actions] 

Finally, the environments will spit the next_states, rewards and if it is done with

next_states, rewards, dones, _ = env.step(actions)

It is at this point I am a bit confused. I think I need to gather immediate experiences, batch altogether and forward it to the agent. My problem is probably with the "gather immediate experiences".

I propose a solution, but I am far from being sure it is a good answer. At each iteration, I think I must take a random number with

nb = random.randint(0, len(n_envs)-1)

and put the experience in history with

history.append(Experience(state=states[nb], actions[nb], rewards[nb], dones[nb]))

Am I wrong? Can you tell me what I should do?

  • $\begingroup$ I think you'd need multiple histories, one for each env $\endgroup$ – Santiago Benoit May 11 '20 at 17:08
  • $\begingroup$ @SantiagoBenoit I answered my own question. Can you tell me if it is a fine way to do it? $\endgroup$ – jgauth May 12 '20 at 14:44
  • $\begingroup$ Looks like the right idea - btw I made an edit to your histories list because it was just repeating the same queue $\endgroup$ – Santiago Benoit May 18 '20 at 5:17
  • $\begingroup$ I have seen what you meant. indeed, if I run histories = [deque(maxlen=5)] * 4 and histories[0].append(1), the output is [deque([1]), deque([1]), deque([1]), deque([1])] which is very annoying. Thanks @SantiagoBenoit! Can you explain that behavior? As you can see, instead I wanted [deque([1]), deque([]), deque([]), deque([])] $\endgroup$ – jgauth May 18 '20 at 10:11
  • $\begingroup$ In Python, multiplication operator on list doesn't create deep copies of objects. Basically what is happening is instead of creating new queues, it is creating references to the same queue. $\endgroup$ – Santiago Benoit May 19 '20 at 4:50
class ExperienceSource():
    def __init__(self, env, agent, reward_steps):
        self.env = env
        self.agent = agent
        self.reward_steps = reward_steps

    def __iter__(self):
        histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]
        states = self.env.reset()

        while True:

            for idx, env in enumerate(self.env.envs):
                action = self.agent.choose_action(states[idx])
                state, reward, done, _ = env.step(action)

                current_rewards[idx] += reward
                histories[idx].append(Experience(state, action, reward, done))

                if len(histories[idx]) == self.reward_steps:
                    yield tuple(histories[idx])

                if done: 
                    yield tuple(histories[idx])
                    state = env.reset()

Be aware that self.reward_steps is simply the value defined by N-1 in the following formula $$Q(s,a) = \sum_{i=0}^{N-1} \gamma^i r_i + \gamma^N V(s_N)$$ and self.env is simply an instance of GymEnvVec class from the question.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.