0
$\begingroup$

I am looking to have a cooperative multi agent reinforcement learning framework where one agent has a discrete action space and another agent has a continuous action space. Is there a way to do this as most papers I have seen will only handle one or the other.

$\endgroup$
2
  • $\begingroup$ you might want to google "parametrized action reinforcement learning" and "parametrized Markov decision process" $\endgroup$
    – Sanyou
    Sep 10 at 3:24
  • $\begingroup$ Thanks Sanyou seems like what I have been looking for. I just didn't really know the correct terms to google were $\endgroup$
    – pd109
    Sep 10 at 17:49
0
$\begingroup$

Seems like what you are looking for is Parametrized RL to train an agent for a Parametrized Markov Decision Process. You can look up both terms to find courses/readings about it.

Anyway, one existing RL framework for it is the Multi-pass Parametrized DQN (MPDQN) which is proposed in this paper, and if google enoug you can even find the author's dissertation on MP-DQN which preliminaries section might help you a lot. In short, the Agent uses Actor-Critic arhitecture in which the Actor will predict the continuous parameters of all actions given the current state, and the Critic will predict the action-value of each action given the states concatenated by the predicted values of the continuous parameters. The final discrete action is chosen by sampling the action-value or using argmax.

In P-DQN, the state concatenated with all the continuous parameters' values are passed to the Critic. However, it is shown that by passing all continuous parameters at once to the Critic, the unused action's parameters will affect the gradient thus affecting the Actor's parameters, which is not expected. Therefore, the parameters of each action are passed separately into the Critic in MP-DQN (unrelated action parameters' are zeroed). With good design, you can implement it so you can pass all the parameters and state in a single pass (multi-pass) as shown in MP-DQN. Other alternative for Parametrized RL is the hybrid-PPO and parametrized DDPG.

Finally, based on my personal experiments, I still have difficulty on controlling the scale of the continuous parameters values prediction so that it stays in a certain predefined range. There are 2 ways to do it, first is using softmax or using gradient inverting (changing the gradients' sign depending on the current predicted parameter values). However, I still eventually produced Actor that output only extreme values (either upper or lower bound) of the continuous parameters.

$\endgroup$
4
  • $\begingroup$ Hi sanyou I have been working my way through the implementation of MPDQN in the github and it seems like they only handle the case where there is one discrete action possible and then many continuous actions. I am in the case where I have a multi-discrete space and continuous actions. Do you think this method would be able to handle this? $\endgroup$
    – pd109
    Sep 17 at 21:01
  • $\begingroup$ yes it can, if you look closely at the agent class definition, there's an intricate masking operation to differentiate which continuous parameters are related to which discrete action. $\endgroup$
    – Sanyou
    Sep 18 at 5:34
  • $\begingroup$ I can see how they are handling the parameterized action spaces in env.action_space where the first space is a Discrete that describes the total number of actions and then the rest are box spaces that hold the high and low value information. However I am not certain how to define my action space in these terms since I have some actions that are just completely discrete. For example I have one action that is just a discrete action between 5 choices and then also 2 other actions that are continuous. All of these actions can be used at a step together. $\endgroup$
    – pd109
    Sep 20 at 22:22
  • $\begingroup$ for the discrete action, just don't use any of the continuous parameter, so that no grad will be backpropagated. While for the continuous actions, that's what the discrete action/label is for, to point what action you choose, and you just so happen to choose a continuous action. Note that, while the continuous actor give you real values, the discrete critic just give you which action to do, right? so no problem with the continuous actions. $\endgroup$
    – Sanyou
    Sep 21 at 6:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.