# How would one implement a multi-agent environment with asynchronous action and rewards per agent?

In a single agent environment, the agent takes an action, then observes the next state and reward:

for ep in num_episodes:
action = dqn.select_action(state)
next_state, reward = env.step(action)


Implicitly, the for moving the simulation (env) forward is embedded inside the env.step() function.

Now in the multiagent scenario, agent 1 ($$a_1$$) has to make a decision at time $$t_{1a}$$, which will finish at time $$t_{2a}$$, and agent 2 ($$a_2$$) makes a decision at time $$t_{1b} < t_{1a}$$ which is finished at $$t_{2b} > t_{2a}$$.

If both of their actions would start and finish at the same time, then it could easily be implemented as:

for ep in num_episodes:
action1, action2 = dqn.select_action([state1, state2])
next_state_1, reward_1, next_state_2, reward_2 = env.step([action1, action2])


because the env can execute both in parallel, wait till they are done, and then return the next states and rewards. But in the scenario that I described previously, it is not clear how to implement this (at least to me). Here, we need to explicitly track time, a check at any timepoint to see if an agent needs to make a decision, Just to be concrete:

for ep in num_episodes:
for t in total_time:
action1 = dqn.select_action(state1)
env.step(action1) # this step might take 5t to complete.
as such, the step() function won't return the reward till 5 t later.
#In the mean time, agent 2 comes and has to make a decision. its reward and next step won't be observed till 10 t later.


To summarize, how would one implement a multiagent environment with asynchronous action/rewards per agents?