4
$\begingroup$

I am new to RL and I'm currently working on implementing a DQN and DDPG agent for a 2D car parking environment. I want to train my agent so that it can successfully traverse the env and park in the designated goal in the middle.

So, my question is: what are the best practices when training an agent for a changing environment?

In my case, my goal is that a car can randomly spawn every episode anywhere in the dark grey-ish area and always successfully parks in the middle. My problem, in this case, is that if I for example train the agent from only one specified location, it usually won't know how to perform if it's spawned somewhere else.

I also tried making it so that the car starting location gets randomly updated every N step, but unfortunately came to no success.

It may be possible that I've not trained for long enough and with a sufficient number of steps in between the "position resets", but I still want to ask if there are any general practices in the cases like this?

Seen here

$\endgroup$

2 Answers 2

5
$\begingroup$

I am correct in my understanding that you only provide the agent with the state of the car, i.e. a global x and y position, its angle, velocity, and steering angle?

How does the agent know that it is coming closer to the goal if it is not provided with information about where the goal is? Without this observation of the goal, the agent is operating blindly. That explains why it is so difficult for the agent to reach the goal and impossible when you randomize the starting position.

If my assumptions are correct, the agent takes random actions which are unlikely to reach the goal, but due to the law of large numbers after enough episodes, the agent will reach the goal at random and it can learn to remember this path if given enough reward. But if you then randomize the starting position the agent cannot apply the knowledge it has learned previously because the sequence of actions to reach the goal would now be different. Essentially, there is no correlation between what goal you want the agent to achieve and your state and action space.

To circumvent this problem, I suggest you add additional state information, here are a few suggestions:

  • The global x and y position of the goal
  • A distance measure measuring the distance from the agent and to the goal. Either the Euclidean distance or Manhattan distance.
  • Both of the above

I also support the suggestion of Elfurd: "Training in steps can be useful". This is called curriculum learning and the idea is to present easier training examples to the agent at the beginning of training and steadily increase the difficulty of the environment. In turn, the agent will reach the goal in the easier environments, obtain some reward, and learn. It can then apply what it has learned in the more advanced environments once it progresses through the curriculum.

In your environment, this could be as simple as decreasing the size of the gird world in early training. Or you could spawn the agent close to the goal so that the agent is more likely to reach the goal with just a few random actions, alternatively, you can also randomize the goal close to the starting position of the agent if it has to start from a specified position and then increase the distance to where the goal is sampled.

$\endgroup$
3
  • 1
    $\begingroup$ From the description it looks like the parking space is always at 0,0 $\endgroup$ Mar 5, 2022 at 9:08
  • 1
    $\begingroup$ Yeah, it seems so. But I imagine they want to be able to start from any arbitrary position and end at any arbitrary position in the environment. $\endgroup$
    – Lars
    Mar 5, 2022 at 9:45
  • 1
    $\begingroup$ Thank you. As you mentioned, I was missing additional state information, namely the distance from the agent to the goal. With this agent is able to learn a good policy faster. I will also try experimenting with curriculum learning to see if the agent will have an easier and faster time learning a good policy. $\endgroup$ Mar 7, 2022 at 20:59
1
$\begingroup$

My guess is that you haven't trained long enough, but there are things that can be done to possibly accelerate learning.

It depends on what you want the policy to do in the final version. If you want it to be able to be spawned at a random position on the map and park in some other (random) position on the map, then that is how you should train it.

Training in steps can be useful, for example training with a fixed start and terminal point, then randomize the parking spot, then randomize the starting position. In general, giving the agent fully randomization will take longer than setting multiple, sequential goals.

$\endgroup$
2
  • $\begingroup$ Tnx for the tip. I will try training the model for longer. The reason I wasn't sure before was that when I tried it (randomizing spawn location every N step), whenever the agent changed position, the loss exploded to an absurd number like 4mil+, it did go down eventually but didn't learn anything. I will try increasing the number of total steps and a reasonable amount of steps before randomizing the spawn location. $\endgroup$ Mar 4, 2022 at 19:16
  • 1
    $\begingroup$ It could be a combination of more training and the right parameters. When you expand the observation space, the problem becomes more difficult and the algorithms needs to try harder to learn and explore. The main thing I have learned in ML is it is more like an experimental science, rather than a logical math-based problem. Even though the learning is built on some basic math results. $\endgroup$
    – Elfurd
    Mar 8, 2022 at 12:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .