1
$\begingroup$

When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that to apply batching to an on-policy method you have to have multiple parallel environments?

As an extension of this, if only one environment is available when using on-policy methods, does that mean batching isn't possible? Doesn't that limit the power of such algorithms in certain cases?

$\endgroup$
1
  • $\begingroup$ Here, here and here are 3 related questions. $\endgroup$ – nbro Oct 12 '20 at 12:56
0
$\begingroup$

We don't need multiple environments. On-policy algorithms require that new training samples are collected with the newest policy, so we can't use an experience buffer. However we can use the newest policy to collect multiple samples, even over multiple epochs, before updating the weights. This update can be a batch update.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.