1
$\begingroup$

I coded some deep RL algorithms (DQN and SAC) with tf2/keras to solve an environment where a vehicle needs to follow the track and avoid crashing into one other vehicle (there is only one other vehicle). Whatever I do, the agent is able to follow road in one way or another but nearly always crashes into other vehicle. I use some kinematics information for observation. (the agent only controls steering)

My observation is the kinematics of the agent and the other vehicle. This includes coordinates, velocities, trigonometric headings, lateral and longitudinal offsets to the closest lane, and angular offsets to the lane.

A reward function could be defined to quantify how close is agent to the center. A negative reward (-1) is provided if the agent crashes. A negative reward is provided if the agent runs out off road.

If there is a crash or if the agent goes off-road, the episode is done.

With this information agent is able to follow the road but as soon as it reaches the other vehicle, it crashes into it. I trained 1000 episodes.

What did i try?

  • I ran my code in different environments and it works. Code structure is not a problem.
  • I added last actions (t and t-1) info to observations.
  • Hyper parameter tuning.
  • Changed crashing reward to different values.
  • Used Stable-Baselines3's PPO algorithm. (Had the same problem.)
  • Added the distance and angular between vehicles to the observation space.
  • I slowed down the discovery rate reduction in DQN.
  • Used PER as buffer in DQN.

None of these solved the crashing problem. Does anyone have any suggestions or ideas to solve this problem?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ What is your reward function for lane closeness? Is it always positive? What is typical return for completed episodes? $\endgroup$
    – Neil Slater
    Commented Nov 7, 2023 at 8:24
  • $\begingroup$ Lane closeness reward is always positive, it's proportional to proximity to lane center. There are two lanes. Since environment is a loop, episodes ends only if vehicle crashes or goes off-road and each situation has negative rewards. $\endgroup$
    – rafiqollective
    Commented Nov 7, 2023 at 11:49

1 Answer 1

Reset to default
1
$\begingroup$

There are a few things I think worth looking into:

  • Is avoiding the other car actually possible in the environment? You may inadvertently be giving the agent a choice of crashing in lane, or going off road once it gets close to the other vehicle. If you have a human control interface you may be able to test how difficult avoiding the other car is.

  • If interactions with the other car are much less common than interactions with road bends, then you would expect the agent to require more examples before it finally figures out correct behaviour. You may also need to adjust learning rates and related params (epsilon decay) to avoid the steering behaviour becoming almost frozen by the time collisions are becoming possible.

  • One thing you could try is randomly varying how close a collision is at the start, including plenty of episodes that start with a collision imminent but avoidable.

  • Check your feature engineering:

  • Are any values related to the other car becoming large magnitude for a neural network? Most values should be in range $[-2,2]$ (with fixed training datasets you can aim for a standard deviation of $1$, but usually you have to compromise to within a range in reinforcement learning because you don't know the distribution)

  • If you have a visualization that shows the env, add a dump of current state and predicted values. Watch it and verify that your features are doing what you expect. One mistake in a key feature can change the appearance of the collision event for the agent from predictable (and avoidable) to apparently random.

One clue as to why the agent is crashing is to look at predicted values close to a collision. If predicted values are low and close to correct, then the agent "knows" it is about to crash, but doesn't know correct behaviour to avoid it. If the predicted values are high, then the agent is not correctly interpreting the state. That could be a training issue or perhaps feature engineering.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for sharing your opinions with me. I see what you mean. Environment actually has two lanes so avoiding is not difficult. I will try what you suggested. Thank you a lot. $\endgroup$
    – rafiqollective
    Commented Nov 7, 2023 at 12:56
  • $\begingroup$ @rafiqollective: My understanding of the environment is that the implementation is like a game with physics engine. And the car cannot change its speed, only steer. So depending on what physics applies to steering, and how the collision detection is done, it is possible to have a problem with this environment where your goal is not realisable, or very hard (e.g. perhaps collision detection is tooo wide, or in practice the agent's car has to steer into alternative lane a long way ahead of other car, and has to predict what lane the other car is in from cartesian co-ordinates) $\endgroup$
    – Neil Slater
    Commented Nov 7, 2023 at 13:02
  • $\begingroup$ Yes you get the environment very well. When you say it like that I'm afraid the environment may be too complicated for the agent to solve. I try feature engineering as you suggested and change reward function. I hope it works. @NeilSlater $\endgroup$
    – rafiqollective
    Commented Nov 7, 2023 at 13:15
  • $\begingroup$ @rafiqollective The environment does not sound too complex for RL, but you may want to rethink features or training routine to address your collision issue. And if possible do try to assess what the correct behaviour is, by e.g. playing the game yourself - that will give you more clues on what the agent should know and react to in order to succeed. $\endgroup$
    – Neil Slater
    Commented Nov 7, 2023 at 13:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .