I've implemented A2C. I'm now wondering why would we have multiple actors walk around the environment and gather rewards, why not just have a single agent run in an environment vector?

I personally think this will be more efficient since now all actions can be calculated together by only going through the network once. I've done some tests, and this seems to work fine in my test. One reason I can think of to use multiple actors is implementing the algorithm across many machines, in which case we can have one agent on a machine. What else reason should we prefer multiple actors?

As an example of environment vector based on OpenAI's gym

class GymEnvVec:
    def __init__(self, name, n_envs, seed):
        self.envs = [gym.make(name) for i in range(n_envs)]
        [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]

    def reset(self):
        return [env.reset() for env in self.envs]

    def step(self, actions):
        return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))
  • 1
    $\begingroup$ Can you clarify what you mean by an "environment vector"? $\endgroup$ Mar 31, 2019 at 4:35
  • $\begingroup$ Hi @PhilipRaeisghasem, please refer to this for an example. $\endgroup$
    – Maybe
    Mar 31, 2019 at 9:13
  • $\begingroup$ @Maybe Where did you take the code related to GymEnvVec? $\endgroup$
    – jgauth
    May 11, 2020 at 13:46

2 Answers 2


I believe if you run a single agent in multiple parallel environments many times you will get similar actions in similar states, the reason behind multiple agents is that you will have different agents with different parameters and you can also have different explicit exploration policies so your exploration will be better and you will learn more from environment (see more state space). With single agent you can't really achieve that, you would have a single exploration policy, single parameter set for the agent and most of the time you would be seeing similar states (at least after a while). You would be speeding up your learning process but that's just because you're running multiple environments in parallel (compared to the regular actor-critic or Q-learning). I think quality of learning would be better with multiple different actors.

  • $\begingroup$ Thanks for clarifying my doubts :) $\endgroup$
    – Maybe
    Mar 31, 2019 at 9:15
  • $\begingroup$ In addition, the multiple agents approach is used for similar reasons to DQN's experience replay - it helps to reduce correlation within training batches. This can be important for stability of a neural network within the A2C framework. $\endgroup$ Mar 31, 2019 at 11:40
  • $\begingroup$ Hi, I still feel confused. In general, A2C is implemented with an on-policy policy-gradient algorithm, If we use different parameters for different agents, then the learning is not on-policy at all. Why would we do that? BTW, @NeilSlater, I cannot see how multiple agents reduce correlation in the way multiple environments do not. Could you please shed some light on it? $\endgroup$
    – Maybe
    Apr 3, 2019 at 1:38
  • $\begingroup$ @SherwinChen: There is no difference between "multiple agents" and "multiple environments" by default, unless you are deliberately connecting the agents in the former (e.g. forcing them to the same state on each time step) or deliberately changing the environments in the latter (e.g. changing the state transition rules). Neither of those things are suggested here. The point is to reduce correlation. In your question you suggest collecting and using data for the variations sequentially - a training batch of your data will be less i.i.d. than a batch which collects from multiple sources. $\endgroup$ Apr 3, 2019 at 6:43
  • $\begingroup$ @NeilSlater, thanks for responding, but I'm sorry that I cannot really understand what you were trying to convey at the end: "In your question you suggest collecting and using data for the variations sequentially - a training batch of your data will be less i.i.d. than a batch which collects from multiple sources." I've updated my question to include a simple environment vector, I cannot see why the data will be less i.i.d. than that collected by multiple agents. I'm looking for your response $\endgroup$
    – Maybe
    Apr 3, 2019 at 7:05

When you run multiple actors you actually make copies of your current agent and run these copies in non-vector environments. The environments are usually also the same.

In the case of Atari, yes of course, using a vectorized environment is much more efficient. But if you want to train an agent to play DotA, or drive a car, then there is no vectorized environment.

Running different actors instead of copies of the same agent is not really possible with on-policy algorithms. You have to collect data using one and the same policy, otherwise it is not on-policy.

A slight exception to this is the case of asynchronous training. In this case you start with the same copies of your agent, but update these copies asynchronously, and only once in a while sync the policies. However, you still have to make sure the actors' policies don't deviate too much from one another. But note that the synchronous approach maybe performs better than the asynchronous. And again, in case you can vectorize your environment I don't really see the need to do this.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .