0
$\begingroup$

I would be grateful for some guidance on a RL problem I am trying to solve where multiple RL agents use a common/global policy at the initial state of an episode in the RL Environment, and then update this common/shared policy once the episode is completed.

Below is an example of the problem scenario:

  • An alert triggers a RL agent to execute a "episode" in the Environment
  • Multiple alerts (e.g., episodes) can occur at the same time, or, one alert may still be being processed (e.g., the episode has not finished) before another alert is triggered (e.g., another episode begins).

Below are the conditions of the Environment and desired behaviour of the RL Agent:

  • Multiple episodes can run at once (e.g., another episode starts before another finishes).
  • For each episode a "instance" of the RL agent uses the latest version of a common policy.
  • After each episode the RL agent updates the common policy.
  • Common policy updates are "queued" using versioning in code to prevent race conditions.

Q: How can multiple RL agents in this case use a common policy at the beginning of an episode and then update a common policy after completing it? - All I have found are discussions related to Q-Learning, where agents can update a shared Q-table, or later update a "global" Q-table without any examples of how this can be achieved and whether there are also methods for other approaches such as TD rather than only Q-Learning, for example

Q: Does this sound like a traditional multi-agent scenario, at least conceptually? If so, how might one go about implementing this, any examples would be really helpful.

Any help on this is greatly appreciated!

EDIT:

Since doing more investigation I have found this reference on Mathworks: Link, which is similar to the above problem, but not exact.

$\endgroup$
0
$\begingroup$

This sounds like distributed RL, and most of the work goes in building the distributed system; the actual RL part is just a DQN (+some tricks from Rainbow DQN). NB: multi-agent RL arises when the agents interact with each other in the same environment (like Hanabi the card game), while in this case we have multiple agents that collect experiences in parallel. Here's a possible design from the Ape-X paper: enter image description here

$\endgroup$
3
  • $\begingroup$ Thanks @kaiwenw for your response and for reference! From my continued reading I was also leaning to towards this definition of dsitributed RL as well. In this case, does distributed RL assume parallel episodes "start", with different learners, at the same time, or is that just an aspect of the design, and not related? For example, where one episode starts with a learner, then say a minute later another episode starts with another new learner (with the original still running), and they are both updating a common policy. $\endgroup$ – RL_NOOB Nov 26 '20 at 20:25
  • $\begingroup$ Excuse me, I meant actor, not learner in the above comment. $\endgroup$ – RL_NOOB Nov 26 '20 at 22:04
  • $\begingroup$ @RL_NOOB everything can be asynchronous, and should be to minimize wait times; you can have some "master", perhaps the learner, decide when to share weights with the actors, or you can have a separate weight server that actor threads will periodically fetch from (in that case you have no direct interaction between learner and actor threads) $\endgroup$ – kaiwenw Nov 26 '20 at 23:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.