I would be grateful for some guidance on a RL problem I am trying to solve where multiple RL agents use a common/global policy at the initial state of an episode in the RL Environment, and then update this common/shared policy once the episode is completed.
Below is an example of the problem scenario:
- An alert triggers a RL agent to execute a "episode" in the Environment
- Multiple alerts (e.g., episodes) can occur at the same time, or, one alert may still be being processed (e.g., the episode has not finished) before another alert is triggered (e.g., another episode begins).
Below are the conditions of the Environment and desired behaviour of the RL Agent:
- Multiple episodes can run at once (e.g., another episode starts before another finishes).
- For each episode a "instance" of the RL agent uses the latest version of a common policy.
- After each episode the RL agent updates the common policy.
- Common policy updates are "queued" using versioning in code to prevent race conditions.
Q: How can multiple RL agents in this case use a common policy at the beginning of an episode and then update a common policy after completing it? - All I have found are discussions related to Q-Learning, where agents can update a shared Q-table, or later update a "global" Q-table without any examples of how this can be achieved and whether there are also methods for other approaches such as TD rather than only Q-Learning, for example
Q: Does this sound like a traditional multi-agent scenario, at least conceptually? If so, how might one go about implementing this, any examples would be really helpful.
Any help on this is greatly appreciated!
Since doing more investigation I have found this reference on Mathworks: Link, which is similar to the above problem, but not exact.