3
$\begingroup$

To make A2C into A3C you make it asynchronous. From what I understand the 'correct' way to do that is to thread off workers with a copy of the policy and critic, and then return the state/action/reward tuples to the main thread, which then performs the gradients updates on the main policy and critic, and then repeat the process.

I understand why the copying would be necessary in a distributed environment, but if I were to always run it locally then could I just perform the updates on a global variable of the policy and critic, i.e. avoid the need for copying? Provided the concurrency of the updates was handled correctly, would that be fine?

Improve this question
$\endgroup$
3
  • $\begingroup$ I don't think it is possible. If the global net updates every N iterations but local net (each agent) updates every M iterations and each agent runs asynchronously, having one copy of the weights is not possible since each agent might need a different copy of the weights. $\endgroup$
    – Phizaz
    Mar 3, 2019 at 6:19
  • $\begingroup$ The global net would be updated every M iterations also. To be clear; the worker would need to 'freeze' a copy of the policy/critic in order to do the updates, it would just get an immediate/online copy through the global variables. The only difference, as I see it, is that each worker has direct access to global variables for the policy/critic to perform the updates, rather than needing to return out with the S, a, r tuples for the main handler thread to do the updates. As I say, I understand the need for that in distributed, but I don't see why it'd be needed on a purely local environment $\endgroup$
    – BigBadMe
    Mar 3, 2019 at 7:14
  • 1
    $\begingroup$ If you allow each worker to share the global net and update on it live, it is possible to do with only one copy of the network. This is called "Hogwild" fashion. I have seen this in Pytorch tutorial itself. $\endgroup$
    – Phizaz
    Mar 3, 2019 at 11:45

1 Answer 1

Reset to default
0
$\begingroup$

You might want to check out this paper relating to Phizaz's comment: Asynchronous Methods for Deep Reinforcement Learning (specifically search for Hogwild).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you. I've since discovered that A2C can still have multiple environments/agents to garner experience more quickly (as you would in A3C), but the difference being that A2C waits for all agents to complete their experiences and apply gradient changes across all agents' batches, whereas A3C performs the gradient updates as soon as it has collected the specified size of experience data for that one agent. The benefit of multi-agent A2C is the agent is always working with the most up-to-date policy, and possible speed improvements if you use a GPU to combine the batches into one large batch. $\endgroup$
    – BigBadMe
    Mar 25, 2019 at 15:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .