2
$\begingroup$

On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-policy algorithms, like SAC and TD3.

Is it because the replay memory is kind of a substitute for the parallel data sampling in the on-policy algorithms? Can off-policy algorithms benefit from the parallelization?

Ray RlLib says the following for SAC and TD3 regarding the number of workers for collecting samples

This only makes sense to increase if your environment is particularly slow to sample

$\endgroup$

1 Answer 1

2
$\begingroup$

From the point of view of someone developing an in-house DRL lib and working on extremely CPU-intensive environments (usually large finite element-based simulations that can require several hours to unroll a single episode), yes it can.

What I do is that I have $n$ parallel environments unrolling simultaneously and collecting transitions, and I perform $n$ updates after each parallel step (i.e. all $n$ parallel envs have collected one transition), in order to keep a 1/1 ratio between stepping and training. I believe it is okay from a theoretical point of view, although I'm not 100% certain. From a practical point of view, it has worked well so far.

The difference between offline and online collection is that the sampling/training procedures are very different, as in online methods you usually sample a large buffer of samples before performing several epochs of training. Yet, I am curious about what would happen if one was to perform the same for, say, TD3: sample a large buffer of $m$ transitions, store them in the replay buffer, and then perform $m$ updates to keep the 1/1 ratio.

$\endgroup$
2
  • $\begingroup$ Have you ever tried to adjust the batch size by multiplying the original batch size for one environment by the number of parallel environments instead in order to maintain 1/1 ratio between stepping and training? I’ve seen one such GitHub implementation of parallelization of off policy algorithms. Obviously this may lead to relatively huge batch sizes; however Open AI is using batch size of tens of millions as reported here openai.com/blog/science-of-ai. But I believe they are applying this in on policy algorithms (since DOTA 2 bots were trained with PPO). $\endgroup$
    – Mika
    Commented Sep 6, 2022 at 2:55
  • $\begingroup$ @Mika thanks for accepting the answer. I'm not certain to understand the follow-up question. In sequential, I perform one gradient step at every time-step by sampling one mini-batch of size $b$ from my replay buffer. In parallel, I still use a mini-batch of size $b$, but I collect $n$ samples in parallel, so I perform $n$ gradient steps at every time-step by sampling $n$ different mini-batches from my replay buffer. What you suggest is to perform $n$ gradient steps, but using mini-batches of size $nb$ ? $\endgroup$ Commented Sep 13, 2022 at 7:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .