I'm modeling a problem using Reinforcement Learning (RL). Formally, I have two agents: one of them is the one that I have to program and model, the other one is unpredictable (random). With unpredictable, I mean that I can't even define a set of possible actions for it.

Given that, I thought of modeling the scenario as a single-agent scenario in which the environment changes even without the intervention of my agent. Can I do it?

Reformulating: in RL, should the environment be modified (and then change) only by the agent in a single-agent scenario? Is it possible that even if the agent makes no action the environment changes or should I add some entity (the random agent) to my model?

I tried to find the answer in the book "Reinforcement Learning: An Introduction", 2nd edition, by Sutton and Barto, but I found nothing answering the question in the introductory chapters (1 and 3). With "I found nothing", I mean that the authors neither say that I can do it, nor that I can't.

  • $\begingroup$ For the second agent, have you been set any task such as predicting its expected return? How are time steps arranged - do both agents act at once, or take turns, or is it more complicated? If you cannot define the action set of the second agent, can you at least still measure the changes it makes to the environment during training? $\endgroup$ Mar 7 at 20:07
  • $\begingroup$ @NeilSlater following the order in which you asked: - no task set for the second agent - random time steps (I could see one action of one agent, the n actions of the other, m of the first and so on) - I cannot see all the changes made by the second agent $\endgroup$ Mar 8 at 17:34
  • $\begingroup$ Thanks for the updaate. When you say "cannot see the changes", does that mean the state includes hidden information (related to what the other agent did) that is important and can affect the success of the first agent? E.g. is it like a poker game where you don't know how many draws the other player has made? Or is everything you need to know for success observable in the state description? E.g. is it like Blackjack where you know your and the dealer's draws, but don't know what any other players have done (so you can generally ignore whatever cards they have drawn)? $\endgroup$ Mar 8 at 18:26
  • $\begingroup$ Thank you for your interest. The model is about a defender of a network and an attacker of the network, so the attacker may be inside the network and it could have modified something, but the defender may not be aware of it. In other words, the other agent (attacker) could have acted or not - if it made an action, I can't be 100% sure to see that it acted and what it did. $\endgroup$ Mar 9 at 9:58
  • $\begingroup$ So, to clarify, the observer can see a state that might include network stats, machine stats etc, but has no idea whether any of those stats have been influenced by the attacker, who is in essence invisible? This is more like Poker, unfortunately. Is the first agent supposed to actively defend against adversarial attacks (that it cannot see), or just raise the alarm or simply shut services down? $\endgroup$ Mar 9 at 11:28


You must log in to answer this question.

Browse other questions tagged .