In an episodic training of an RL agent, should I always start from the same initial state or I can start from several valid initial states?
For example, in a gym environment, should my env.reset() function always resets me to the same start state or it can start from different states at each training episode?
It depends on the task the agent is trying to learn and of course on the environment constrains.
In an Atari game agents have a pre-fixed starting point because that's part of the games rules, so I would say that this is enough of a justification to make each simulation start from that starting point. Moreover, you have to pay attention to the kind of reward function you're using, for example (a really dumb one just to give a grasp of the concept) if you're rewarding the agent depending on how close it gets to the ending point rather than how far it goes from the starting point, the agent might end up having huge rewards only because it respawned close to end point, which would be an artefact and not a fair reward for a good action choice. Other situations in which it does not really make sense to select a random starting point might be dialogue systems, in which you know that a conversation start with greetings, therefore it would not be logical to make an agent starting with a random question or sentence (the space in these case would be made of different dialogue acts).
There are anyway situations in which the environment allows a random selection of the start location of the agent. In this paper for example, an agent was trained to escape from a maze, and both, the exit point and the initial location of the agent where randomly selected at each iteration. This was partially due to research reasons, the authors were analysing the possibility to train policies more complex than a simple 'space memorisation' one but anyway, conceptually there's nothing wrong in this situation in selecting a random starting point. Other tasks could be following or escaping from a specific object, again, as long as the reward function is properly designed, even with a random starting location at the end the agent would anyway learn to move faster when close to the target (toward or in the opposite direction depending on the task). Actually, in this situation I think that a random initial location would have potential benefits, like avoiding the agents to learn biases due to external biases of the environment (like parts of space with less random obstacles than others).
It is your choice.
This can even be different between training and target system. The approach called "exploring starts" chooses a random start state (and action if you are assessing a deterministic policy for action values).
In general, if you don't have a reason to pick exploring starts, you should aim for your env.reset() function to put the environment into a state drawn from the distribution of start states that you expect the agent to encounter in production. This will help if you are using function approximation - it will mean that the distribution of training data will better match the distribution seen in production, and approximators can be sensitive to that.
In some cases, such as policy-gradient methods, your cost function will be defined in terms of expected return given a start state distribution, so at least during assessment you will want a env.reset() function that matches that target start distribution.
It is still OK to have different distributions for start states for training and assessment, and might be worth investigating as a hyperparameter for training. For instance if the training start state distribution can pick states that are hard to get to randomly otherwise, it may help the agent to find any of those states that are high value.
