What is the effect of parallel environments in reinforcement learning?

Question

Do parallel environments improve the agent's ability to learn or does it not really make a difference? Specifically, I am using PPO, but I think this applies across the board to other algorithms too.

Answer 1

Do parallel environments improve the agent's ability to learn or does it not really make a difference?

Yes they can make a difference. There are two ways improvement is seen:

• Collecting data from multiple trajectories at once reduces correlation in the dataset. This improves convergence for online learning systems like neural networks, which work best with i.i.d. data.

• Data collection is faster overall, which improves clock time to obtain the same result. This may make better use of other resources too.

Of the two, the first improvement is important for stability, although it can be emulated by running multiple episodes - or restarting from multiple starting points - between batch learning updates.

Specifically, I am using PPO, but I think this applies across the board to other algorithms too.

It does apply to PPO, but the first improvement does not apply across the board. These things need to be true for environment run in paralell to be help with stability:

• Using an on-policy method, or where experience replay is not an option.

• Using a function approximator for policy and/or value function.

A lot of policy gradient methods match this, including PPO, A3C, REINFORCE. However, for an off-policy method like DQN, the main benefit will be faster data collection.

These effects are discussed in sections 1 and 4 of the paper Asynchronous Methods for Deep Reinforcement Learning which introduced A3C - thanks to DeepQZero for that reference.