6
$\begingroup$

There are multiple ways to implement parallelism in reinforcement learning. One is to use parallel workers running in their own environments to collect data in parallel, instead of using replay memory buffers (this is how A3C works, for example).

However, there are methods, like PPO, that use batch training on purpose. How is parallelism usually implemented for algorithms that still use batch training?

Are gradients accumulated over parallel workers and the combined? Is there another way? What are the benefits of doing parallelism one way over another?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
3
$\begingroup$

OpenAI have a post on that: https://openai.com/blog/openai-five/

They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed for batches of 4096 observations which are then averaged.

PPO is actually designed to allow this kind of parallelisation as it uses trajectory segments with a fixed size of $T$ to collect data, e.g. 60 seconds for OpenAI Five, where $T$ is supposed to be "much less than the episode length" (p.5 of PPO paper).

Improve this answer
$\endgroup$
1
  • $\begingroup$ So does it update each worker agent after each policy update? And is this at all related to 'batched evironments' like can be seen here $\endgroup$
    – profPlum
    Commented Jan 17, 2022 at 22:18
1
$\begingroup$

The paper Dota 2 with Large Scale Deep Reinforcement Learning goes into greater detail than the initial blog posts.

They call their distributed training framework Rapid, which is also used in some of their robotics work, such as the paper Learning Dexterous In-Hand Manipulation, where they discuss a smaller scale deployment of Rapid (as compared to Dota2/OpenAI V) in section 4.3.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .