1
$\begingroup$

Say I have a simple multi-agent reinforcement learning problem using vanilla policy gradient methods (i.e. REINFORCE) that is currently running with one network per agent. If I can say that each of my agents:

  • are all of the same class
  • have ~equivalent environmental contexts (on average)
  • have no privileged state relative to other agents
  • performs updates equally to all other agents
  • DO use LSTMs (but store and reset memory states separately)
  • DO receive rewards for individual actions based on their individual states

...is it possible to use one network for all agents so as to minimize training time? And if so, how do I combine rewards and generate losses? For instance, if I calculate loss as the -logprob * reward (in the REINFORCE case), could I simply sum or average this over all agents and then backprop accordingly?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Yes, this can be done and is widely applied in recent literature on multi-agent RL, at least with the collaborative setting where agents are optimizing a shared reward. This is also known as parameter sharing between agents. Note that, even if the agents share the parameters in their policy networks, you can still get different behavior if the agents get different inputs, e.g., perceive different observations, or have different LSTM states.

In terms of the loss function, no changes are required compared to the case where each agent has its own network with private parameters, as long as you make sure the network with shared parameters correctly processes the individual inputs of each agent.

A concrete example is mentioned here - see Appendix C. They also use the trick of appending a one-hot encoding of the agent index with the observation to enable learning of policies that differ based on agent identity. Or, if you prefer to see an implementation of the idea in code, check out how agents are defined in this repository.

Improve this answer
$\endgroup$
4
  • $\begingroup$ very helpful answer. I am however skeptical that "no change" is required. If I calc loss for each agent and backpropagate individually, the network is no longer the same after the first weight update (and so the second agent's backprop is now on a different set of weights). I think for there to be no change it would require that the agents take actions and update sequentially, but this would not be valid if they moved in parallel, right? $\endgroup$
    – Josh
    Sep 13, 2022 at 12:17
  • 1
    $\begingroup$ That's a good point, this is true at least for the case of cooperative agents that are optimizing a shared reward - this is also the setting covered by the resources I linked. I've clarified this point. I am not quite sure about the competitive setting where each agent has its own reward function - perhaps parameter sharing generally is not terribly appropriate in such a setting (why would one tell competitors what policy one is using?). $\endgroup$
    – mikkola
    Sep 13, 2022 at 14:25
  • $\begingroup$ In my case the agents are "cooperative" in that they share a team reward, but they have individual rewards related to their individual movement that helps them learn the behavior of the more complex team reward swiftly. I believe in such a case I should be able to backprop all at once either by averaging or summing and can find that out empirically. $\endgroup$
    – Josh
    Sep 13, 2022 at 21:11
  • 1
    $\begingroup$ Adding for relevance of any others reading later, I honestly just averaged the losses and it worked fine. In fact it reduced some of the variance in learning as one might intuitively expect. $\endgroup$
    – Josh
    Sep 17, 2022 at 19:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .