1
$\begingroup$

I am experimenting with MADDPG algorithm implemented in this repo. Since there were only a few agents (2-3) in the implementation (also in the original paper) steps like parameter updates, action prediction, etc. are done in a for loop. I want to increase the number of agents, say 10 or 30, and perform parallelization of the above-mentioned steps for all agents, i.e. I want to avoid for loops like this

for agent_idx in range(n_agents):
    ...
    ...

I tried Python Multiprocessing module with pool.map method but I am getting 'AttributeError: Can't pickle local object ...". Below is code I am running to get a joint action prediction but it results in the error above.

def get_ind_action(i, obs_i):
    return actor_critic[i].act(obs_i) # returns an individual action for a given observation for ith agent

def get_joint_action(obs):
    pool = Pool()
    args_list = [[i, obs[i]] for i in range(n_agents)]
    joint_action = pool.map(get_ind_action, args_list)
    return joint_action
    

Here actor_critic is a list of neural networks of all agents, obs is the joint state observed by the centralized critic but each actor only sees its own state. The algorithm has the following architecture.

enter image description here

$\endgroup$
5
  • 1
    $\begingroup$ Hey, maybe to help someone unfamiliar with MADDPG (like me) to quickly grasp the difficulty of parallelizing it can you clarify that (a) are the agents gonna share info about their parameters at some point? (b) and if any other information is going to be shared among process in the learning phase? (c) are you perhaps planning to parallelize on single node or multi nodes? (d) to answer the error perhaps share short reproducible parts of the code that you add to the code you linked $\endgroup$
    – Sanyou
    Sep 6 at 6:34
  • $\begingroup$ (a) they have identical architecture, i.e. the same number of layers and units in each layer but share no parameters. (b) no information is shared among agents' actor networks. (c) single node. (d) added above $\endgroup$
    – Mika
    Sep 8 at 0:44
  • 1
    $\begingroup$ You have to pass the agent as the argument to the function, because the subprocess do not have the agent in the memory. You might want to pass the actor's state_dict instead of the agent and reinstantiate the agent in the subprocess->load the state_dict -> and predict the action -> return the action $\endgroup$
    – Sanyou
    Sep 8 at 1:27
  • $\begingroup$ Yes, I think actor’s state_dict should be passed, I’ve just tried to pass the agent and I got the same error. Let me give a try. Thanks. $\endgroup$
    – Mika
    Sep 8 at 1:30
  • 1
    $\begingroup$ if you're on linux try other start method (spawn, fork, forkserver) and see the effect on the running time, or even start all processes first and send the state_dict and state to Queue to prevent instantiating processes on every step. $\endgroup$
    – Sanyou
    Sep 8 at 1:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.