I depends on your overall model architecture (and problem specification). As I understand it, you take the observations of all agents together and feed it into one model, a central controller, which then predicts the action per available agent.
I believe that this varying number of applicable observations (depending on the number of currently present agents) is what you mean by changing state spaces?
One option that works in these kinds of settings is to use padding. So, a fixed number of individual observations gets predetermined before training, being set to the average number of agents expected to be present in the environment at any timestep.
Then, during training, local, individual observations of all agents (for the current time step) get stacked together, resulting in an overall observation being fed into central controller.
Then, the controller predicts (concurrently/in a single pass) one action (or multiple, depends on what you want it to predict per agent...) per agent.
In the case that the number of present agents currently (=in a given time step) is smaller than the predetermined number of expected agents, then padding is used to fill the empty observation slots by zero-pseudo-observations. The quality of the predicted actions is consequently only assessed for the present agents, neglecting predictions for absent agents, indicated by their empty observation slots.
If the number of controlled agents happens to exceed the expected maximal number of agents present in the simulation, excess agents are selected at random and randomly controlled for the next time step. Then, this (randomized control of a random selection of some agents) continues until no excess agents are present in the simulation anymore. Alternatively, if available, other reasonable controllers might be used to temporarily steer excess agents.
I have seen researchers doing this in the context of traffic control, where a variable number of cars was controlled by a central controller.
See this paper (Section 3.2) for an example of the aforementioned technique.
Alternatively, there exist approaches where each agent gets a periodically updated local copy of some central controller and then contributes to jointly updating the central controller. This paper shortly summarizes a lot of different options and possibilities to choose from in terms of different RL algorithms, where different options might apply depending on the concrete task at hand. Also, the paper discusses different options with respect to algorithms implemented in the python Reinforcement Learning (RL) library RLlib, which might already provide a suitable implementation for a class of algorithm you want to work with.
Unfortunately, a lot of papers don't reveal explicitly whether their (partly) distributed RL training procedures are more suitable, i.e. converge stably to a good solution, for single-agent environments or equally suitable for multi-agent environments as well.
When it comes to introducing new agents to the simulation, this depends entirely on your concrete problem at hand. In the context of vehicle control, you could position a vehicle at the start of the simulated road, steer it (safely) a few time steps straight ahead (to initialize its observation space) and then hand over control to the RL controller. For other types of agents (e.g. soccer playing robots etc.) that might look entirely different. In that case, you might have them sitting or standing next to the field and then have them enter the play ground or so. So, that really depends on your task at hand.
