How to deal with changing environment in reinforcement learning

Question

I am new to RL and I'm currently working on implementing a DQN and DDPG agent for a 2D car parking environment. I want to train my agent so that it can successfully traverse the env and park in the designated goal in the middle.

So, my question is: what are the best practices when training an agent for a changing environment?

In my case, my goal is that a car can randomly spawn every episode anywhere in the dark grey-ish area and always successfully parks in the middle. My problem, in this case, is that if I for example train the agent from only one specified location, it usually won't know how to perform if it's spawned somewhere else.

I also tried making it so that the car starting location gets randomly updated every N step, but unfortunately came to no success.

It may be possible that I've not trained for long enough and with a sufficient number of steps in between the "position resets", but I still want to ask if there are any general practices in the cases like this?

Answer 1

I am correct in my understanding that you only provide the agent with the state of the car, i.e. a global x and y position, its angle, velocity, and steering angle?

How does the agent know that it is coming closer to the goal if it is not provided with information about where the goal is? Without this observation of the goal, the agent is operating blindly. That explains why it is so difficult for the agent to reach the goal and impossible when you randomize the starting position.

If my assumptions are correct, the agent takes random actions which are unlikely to reach the goal, but due to the law of large numbers after enough episodes, the agent will reach the goal at random and it can learn to remember this path if given enough reward. But if you then randomize the starting position the agent cannot apply the knowledge it has learned previously because the sequence of actions to reach the goal would now be different. Essentially, there is no correlation between what goal you want the agent to achieve and your state and action space.

To circumvent this problem, I suggest you add additional state information, here are a few suggestions:

• The global x and y position of the goal
• A distance measure measuring the distance from the agent and to the goal. Either the Euclidean distance or Manhattan distance.
• Both of the above

---

I also support the suggestion of Elfurd: "Training in steps can be useful". This is called curriculum learning and the idea is to present easier training examples to the agent at the beginning of training and steadily increase the difficulty of the environment. In turn, the agent will reach the goal in the easier environments, obtain some reward, and learn. It can then apply what it has learned in the more advanced environments once it progresses through the curriculum.

In your environment, this could be as simple as decreasing the size of the gird world in early training. Or you could spawn the agent close to the goal so that the agent is more likely to reach the goal with just a few random actions, alternatively, you can also randomize the goal close to the starting position of the agent if it has to start from a specified position and then increase the distance to where the goal is sampled.

Answer 2

My guess is that you haven't trained long enough, but there are things that can be done to possibly accelerate learning.

It depends on what you want the policy to do in the final version. If you want it to be able to be spawned at a random position on the map and park in some other (random) position on the map, then that is how you should train it.

Training in steps can be useful, for example training with a fixed start and terminal point, then randomize the parking spot, then randomize the starting position. In general, giving the agent fully randomization will take longer than setting multiple, sequential goals.