1
$\begingroup$

I've implemented A2C, but I'm now wondering why we have multiple actors walk around the environment and gather rewards, why not just have a single agent run in an environment vector? I personally think this will be more efficient since now all actions can be calculated together by only going through the network once. I've done some tests, and this seems to work fine in my test. One reason I can think of to use multiple actors is implementing the algorithm across many machines, in which case we can have one agent on a machine. What else reason should we prefer multiple actors?

As an example of environment vector based on OpenAI gym

class GymEnvVec:
    def __init__(self, name, n_envs, seed):
        self.envs = [gym.make(name) for i in range(n_envs)]
        [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]

    def reset(self):
        return [env.reset() for env in self.envs]

    def step(self, actions):
        return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))
$\endgroup$
  • 1
    $\begingroup$ Can you clarify what you mean by an "environment vector"? $\endgroup$ – Philip Raeisghasem Mar 31 '19 at 4:35
  • $\begingroup$ Hi @PhilipRaeisghasem, please refer to this for an example. $\endgroup$ – Maybe Mar 31 '19 at 9:13
1
$\begingroup$

I believe if you run a single agent in multiple parallel environments many times you will get similar actions in similar states, the reason behind multiple agents is that you will have different agents with different parameters and you can also have different explicit exploration policies so your exploration will be better and you will learn more from environment (see more state space). With single agent you can't really achieve that, you would have a single exploration policy, single parameter set for the agent and most of the time you would be seeing similar states (at least after a while). You would be speeding up your learning process but that's just because you're running multiple environments in parallel (compared to the regular actor-critic or Q-learning). I think quality of learning would be better with multiple different actors.

$\endgroup$
  • $\begingroup$ Thanks for clarifying my doubts :) $\endgroup$ – Maybe Mar 31 '19 at 9:15
  • $\begingroup$ In addition, the multiple agents approach is used for similar reasons to DQN's experience replay - it helps to reduce correlation within training batches. This can be important for stability of a neural network within the A2C framework. $\endgroup$ – Neil Slater Mar 31 '19 at 11:40
  • $\begingroup$ Hi, I still feel confused. In general, A2C is implemented with an on-policy policy-gradient algorithm, If we use different parameters for different agents, then the learning is not on-policy at all. Why would we do that? BTW, @NeilSlater, I cannot see how multiple agents reduce correlation in the way multiple environments do not. Could you please shed some light on it? $\endgroup$ – Maybe Apr 3 '19 at 1:38
  • $\begingroup$ @SherwinChen: There is no difference between "multiple agents" and "multiple environments" by default, unless you are deliberately connecting the agents in the former (e.g. forcing them to the same state on each time step) or deliberately changing the environments in the latter (e.g. changing the state transition rules). Neither of those things are suggested here. The point is to reduce correlation. In your question you suggest collecting and using data for the variations sequentially - a training batch of your data will be less i.i.d. than a batch which collects from multiple sources. $\endgroup$ – Neil Slater Apr 3 '19 at 6:43
  • $\begingroup$ @NeilSlater, thanks for responding, but I'm sorry that I cannot really understand what you were trying to convey at the end: "In your question you suggest collecting and using data for the variations sequentially - a training batch of your data will be less i.i.d. than a batch which collects from multiple sources." I've updated my question to include a simple environment vector, I cannot see why the data will be less i.i.d. than that collected by multiple agents. I'm looking for your response $\endgroup$ – Maybe Apr 3 '19 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.