I'm trying to design an OpenAI Gym environment in which multiple users/players perform actions over time. It's round based and each user needs to take an action before the round is evaluated and the next round starts. The action for one user can be model as a gym.spaces.Discrete(5) space. I want my RL agent to make decisions for all users. I'm wondering how to take multiple actions before progressing time and calculating the reward.

Basically, what I want is:

obs = env.reset()
user_actions = []
for each user:
obs, reward, done, _ = env.step(user_actions)

So the problem is that I don't immediately know the reward after getting an action since I need to collect all actions before evaluating the round.

I could of course extend actions to include actions of all users in one go. But this would be problematic if I have a really large number of users or it even changes over time, right?

I found these two (1, 2) related questions, but they didn't solve my problem.

  • $\begingroup$ What was the format of action space you used. Was it Discrete or Multi Discrete? and how did you map that action space to your dict for passing into your step function ? $\endgroup$
    – kikram
    Apr 18, 2021 at 11:21
  • $\begingroup$ @kikram See my question and answer below. Here, my actions were discrete and the dict, in my answer, maps agent ID to agent action. $\endgroup$ Apr 19, 2021 at 12:11

1 Answer 1


What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own reward.

I represent my RL agents' actions as dict, containing the RL agent ID as key and its action as value. The different agents may either use the same or a different policy and use this policy and each their own observations (also stored as dict) to compute their actions.

action = {}
for agent_id, agent_obs in obs.items():
    # get the policy of the current agent
    policy_id = self.config['multiagent']['policy_mapping_fn'](agent_id)
    # use the policy and the agent's observations to compute its next action
    action[agent_id] = self.agent.compute_action(agent_obs, policy_id=policy_id)

My environment's step(self, action) function expects action to be such a dict and knows how to handle it. It applies all actions in the dict before calculating each agent's reward and progressing time in the environment. So basically, treating observations, actions, rewards (and info) as dicts solves the problem for me.

For the multi-agent RL approach, I used PPO and the Ray RLlib framework.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .