# How to handle a changing in the Reinforcement Learning environment where there is increasing or decreasing in number of agents?

I'm working in A2C and I have an environment where there is increasing or decreasing in the number of agents. The action space in the environment will not change but the state will change when new agents join or leave the game. I have tried encoder-decoder model with attention but the problem is that the state and the model will change when the number of agents is changing. I also tried this way where they use LSTM to get the Q value for the agent but I got this message

Cannot interpret feed_dict key as Tensor: Tensor Tensor("state:0", shape=(137,), dtype=float32) is not an element of this graph.


or error like this because of changing of state size

ValueError: Cannot feed value of shape (245,) for Tensor 'state:0', which has shape '(161,)'


(1) Are there any reference papers that deal with such a problem?

(2) What is the best way to deal with the new agents that join or leave the game?

(3) How to deal with the changing of state space?

I depends on your overall model architecture (and problem specification). As I understand it, you take the observations of all agents together and feed it into one model, a central controller, which then predicts the action per available agent.

I believe that this varying number of applicable observations (depending on the number of currently present agents) is what you mean by changing state spaces?

One option that works in these kinds of settings is to use padding. So, a fixed number of individual observations gets predetermined before training, being set to the average number of agents expected to be present in the environment at any timestep.

Then, during training, local, individual observations of all agents (for the current time step) get stacked together, resulting in an overall observation being fed into central controller.

Then, the controller predicts (concurrently/in a single pass) one action (or multiple, depends on what you want it to predict per agent...) per agent.

In the case that the number of present agents currently (=in a given time step) is smaller than the predetermined number of expected agents, then padding is used to fill the empty observation slots by zero-pseudo-observations. The quality of the predicted actions is consequently only assessed for the present agents, neglecting predictions for absent agents, indicated by their empty observation slots. If the number of controlled agents happens to exceed the expected maximal number of agents present in the simulation, excess agents are selected at random and randomly controlled for the next time step. Then, this (randomized control of a random selection of some agents) continues until no excess agents are present in the simulation anymore. Alternatively, if available, other reasonable controllers might be used to temporarily steer excess agents.

I have seen researchers doing this in the context of traffic control, where a variable number of cars was controlled by a central controller. See this paper (Section 3.2) for an example of the aforementioned technique.

Alternatively, there exist approaches where each agent gets a periodically updated local copy of some central controller and then contributes to jointly updating the central controller. This paper shortly summarizes a lot of different options and possibilities to choose from in terms of different RL algorithms, where different options might apply depending on the concrete task at hand. Also, the paper discusses different options with respect to algorithms implemented in the python Reinforcement Learning (RL) library RLlib, which might already provide a suitable implementation for a class of algorithm you want to work with.

Unfortunately, a lot of papers don't reveal explicitly whether their (partly) distributed RL training procedures are more suitable, i.e. converge stably to a good solution, for single-agent environments or equally suitable for multi-agent environments as well.

When it comes to introducing new agents to the simulation, this depends entirely on your concrete problem at hand. In the context of vehicle control, you could position a vehicle at the start of the simulated road, steer it (safely) a few time steps straight ahead (to initialize its observation space) and then hand over control to the RL controller. For other types of agents (e.g. soccer playing robots etc.) that might look entirely different. In that case, you might have them sitting or standing next to the field and then have them enter the play ground or so. So, that really depends on your task at hand.

• Thanks for your answer. Actually, every agent has a different state. Also, feeding the state with random information may affect the learning results. Also, The only changes that happened to the state are the size when the number of agents increases or decreases. However, most of the traffic control papers have one goal a global reward that not much affects the training but in my case, I need every agent to get his local reward. arxiv.org/abs/1908.03761 – i_th Aug 28 '20 at 16:19
• I have another problem where the random in most cases is larger than the state itself and I have tried it but the results are getting worse. I generate the random depends on the state. – i_th Aug 29 '20 at 4:19
• I found the paper I mentioned earlier and clarified the answer a bit. I hope this helps. – Daniel B. Aug 29 '20 at 18:29
• When I said "zero/meaningless random data", I wasn't entirely sure anymore how the authors had described their method in the paper. But now, that I have found the paper, it clearly says they used zero padding. So, I'd recommend sticking to that. Regarding suboptimal performance, that may have also to do with reasons other than just how you realize group control. For example, maybe a different RL algo would perform better. Or trying different exploration strategies might lead to better results etc. So I am not sure whether subopt. performance is still due to the original issue of this thread. – Daniel B. Aug 30 '20 at 11:53
• But actually, I would not be surprised to see results being worse in a condition where the number of agents is variable (given same amount of training). This is simply because it adds to the complexity of the environment each agent has to function in if the number of controlled agents in an environment varies from time to time. It just requires more training to figure out the best policy per number of agents present in the environment if that nr changes (which is all fixed when the nr of agents is fixed). A variable number of agents just 'complexifies' the dynamics of the overall RL system. – Daniel B. Aug 30 '20 at 19:25