3
$\begingroup$

In reinforcement learning, an agent is usually fully autonomous and independent. It executes actions on the environment, but no other agent can control, explore or command this agent.

In multi-agent reinforcement learning, can one agent explore, command or communicate with other agents?

Would communication between agents somehow be different from the reciprocal execution of each other's actions between agents?

|improve this question
$\endgroup$
  • 1
    $\begingroup$ What do you mean by "reciprocal execution of each other's actions between agents"? $\endgroup$ – nbro Jul 15 at 21:15
  • $\begingroup$ "Reciprocal execution" - is it possible that one agent act as the environment of the other agent and hence exposes actions for command itself and similarly the other agent can act as the environment for the first agent. Just explaining what I meant initially. I am still digesting Manuel's answer and I now have my own thoughts, I will detail them later. The solution lies in the notion of the "action" - can action which returns callback (from environment back into action) be an action for reinforcement learning setting? $\endgroup$ – TomR Jul 15 at 22:10
  • $\begingroup$ @TomR What do you mean "hence exposes actions for command itself", do you mean that the first agent (the environment) decides which actions are available for the second agent? If this is your question, then I will remove "In multi-agent reinforcement learning, can one agent explore, command or communicate with other agents?", because this is a sightly different question. $\endgroup$ – nbro Jul 15 at 23:10
1
$\begingroup$

Let me first summarize the question to make sure, that we are talking about the same issue. Reinforcement learning agent #1 is able to solve the inverted pendulum problem. He can execute the correct actions to maintain the pendulum in upright position. Agent #2, which is also an RL agent, doesn't control the environment but it is sending commands to Agent #1. Which means that the agent has to learn how to communicate with a different agent. Agent #1 plays the social role of the environment and the task is to optimal control this environment.

This sounds a bit like a command chain, in which the workload is distributed among different AI entities. Or to explain it easier, it's a hierarchical reinforcement learning issue in which the idea of learning is task is used recursively. Let us go back to the example problem and describe how a meaningful communication would look like. Agent #1 has the ability to maintain the balance of a pendulum. He accepts the following input commands:

  1. run (bring Pendulum upward)
  2. stop (do nothing)

The question for the manager agent is, when to execute such a command. Is it the right time to balance the pendulum or can agent #1 remain in the sleep mode and do nothing? This kind of decision is made on higher hierarchical level. Which means, agent #2 doesn't know how to balance the pendulum itself, but he knows only how to motivate a different computer program.

hierarchicalrl.png

According to the picture, it's mainly a language parsing problem. The social role of each agent depends on the words which he accepts as input. The language space of agent #1 has to do with sending left/right statements to the environment, while the discursive sphere of agent #2 is about activating a task.

|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.