On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-policy algorithms, like SAC and TD3.

Is it because the replay memory is kind of a substitute for the parallel data sampling in the on-policy algorithms? Can off-policy algorithms benefit from the parallelization?

Ray RlLib says the following for SAC and TD3 regarding the number of workers for collecting samples

This only makes sense to increase if your environment is particularly slow to sample


1 Answer 1


From the point of view of someone developing an in-house DRL lib and working on extremely CPU-intensive environments (usually large finite element-based simulations that can require several hours to unroll a single episode), yes it can.

What I do is that I have $n$ parallel environments unrolling simultaneously and collecting transitions, and I perform $n$ updates after each parallel step (i.e. all $n$ parallel envs have collected one transition), in order to keep a 1/1 ratio between stepping and training. I believe it is okay from a theoretical point of view, although I'm not 100% certain. From a practical point of view, it has worked well so far.

The difference between offline and online collection is that the sampling/training procedures are very different, as in online methods you usually sample a large buffer of samples before performing several epochs of training. Yet, I am curious about what would happen if one was to perform the same for, say, TD3: sample a large buffer of $m$ transitions, store them in the replay buffer, and then perform $m$ updates to keep the 1/1 ratio.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .