I've implemented A2C. I'm now wondering why would we have multiple actors walk around the environment and gather rewards, why not just have a single agent run in an environment vector?
I personally think this will be more efficient since now all actions can be calculated together by only going through the network once. I've done some tests, and this seems to work fine in my test. One reason I can think of to use multiple actors is implementing the algorithm across many machines, in which case we can have one agent on a machine. What else reason should we prefer multiple actors?
As an example of environment vector based on OpenAI's gym
class GymEnvVec: def __init__(self, name, n_envs, seed): self.envs = [gym.make(name) for i in range(n_envs)] [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)] def reset(self): return [env.reset() for env in self.envs] def step(self, actions): return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))