OpenAI Gym: Multiple actions in one step

Question

I'm trying to design an OpenAI Gym environment in which multiple users/players perform actions over time. It's round based and each user needs to take an action before the round is evaluated and the next round starts. The action for one user can be model as a gym.spaces.Discrete(5) space. I want my RL agent to make decisions for all users. I'm wondering how to take multiple actions before progressing time and calculating the reward.

Basically, what I want is:

obs = env.reset()
user_actions = []
for each user:
    user_actions.append(agent.predict(obs))
obs, reward, done, _ = env.step(user_actions)

So the problem is that I don't immediately know the reward after getting an action since I need to collect all actions before evaluating the round.

I could of course extend actions to include actions of all users in one go. But this would be problematic if I have a really large number of users or it even changes over time, right?

I found these two (1, 2) related questions, but they didn't solve my problem.

Answer 1

What I was looking for is multi-agent RL, where I have multiple RL agents, each controlling actions of one user. All RL agents/user make an action in each environment step and each get their own reward.

I represent my RL agents' actions as dict, containing the RL agent ID as key and its action as value. The different agents may either use the same or a different policy and use this policy and each their own observations (also stored as dict) to compute their actions.

action = {}
for agent_id, agent_obs in obs.items():
    # get the policy of the current agent
    policy_id = self.config['multiagent']['policy_mapping_fn'](agent_id)
    # use the policy and the agent's observations to compute its next action
    action[agent_id] = self.agent.compute_action(agent_obs, policy_id=policy_id)

My environment's step(self, action) function expects action to be such a dict and knows how to handle it. It applies all actions in the dict before calculating each agent's reward and progressing time in the environment. So basically, treating observations, actions, rewards (and info) as dicts solves the problem for me.

For the multi-agent RL approach, I used PPO and the Ray RLlib framework.