4
$\begingroup$

Do parallel environments improve the agent's ability to learn or does it not really make a difference? Specifically, I am using PPO, but I think this applies across the board to other algorithms too.

$\endgroup$
4
$\begingroup$

Do parallel environments improve the agent's ability to learn or does it not really make a difference?

Yes they can make a difference. There are two ways improvement is seen:

  • Collecting data from multiple trajectories at once reduces correlation in the dataset. This improves convergence for online learning systems like neural networks, which work best with i.i.d. data.

  • Data collection is faster overall, which improves clock time to obtain the same result. This may make better use of other resources too.

Of the two, the first improvement is important for stability, although it can be emulated by running multiple episodes - or restarting from multiple starting points - between batch learning updates.

Specifically, I am using PPO, but I think this applies across the board to other algorithms too.

It does apply to PPO, but the first improvement does not apply across the board. These things need to be true for environment run in paralell to be help with stability:

  • Using an on-policy method, or where experience replay is not an option.

  • Using a function approximator for policy and/or value function.

A lot of policy gradient methods match this, including PPO, A3C, REINFORCE. However, for an off-policy method like DQN, the main benefit will be faster data collection.

These effects are discussed in sections 1 and 4 of the paper Asynchronous Methods for Deep Reinforcement Learning which introduced A3C - thanks to DeepQZero for that reference.

$\endgroup$
2
  • 1
    $\begingroup$ Why only for on-policy and why for no exp.replay? $\endgroup$
    – ddaedalus
    May 23 at 22:47
  • 1
    $\begingroup$ @ddaedalus Experiance replay is only feasible for off-policy, because older records will have been made with a different policy - off-policy can handle that, on-policy cannot. If you have experience replay, you already have the benefit of improved convergence from parallel trajectories, because you already have closer to i.i.d. from sampling the experience replay memory. $\endgroup$ May 24 at 6:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.