I have built a custom RL environment with gym, which simulates the RL vehicle and potential vehicles in front of the RL vehicle as well as traffic lights (including their state; red, yellow, green). I trained a PPO agent from stable_baselines3 without considering setting of hyperparameters and the agent learned to follow the vehicle in front of it without crashing. However it does not learn to stop at red lights after extensive training.

I tried training it without surrounding vehicles to get more interactions of the RL vehicle with traffic lights and this helped the agent to learn stopping a red light. However when I then continue training of the agent in a new environment with surrounding traffic, the agent again un-learns stopping at red lights.

I am still a novice with RL and do not understand as to why this happens and what I can do here. Should I set hyperparameters? Or try a different model? Or should I exchange the default policy of the PPO model?


1 Answer 1


The way I dealt with it was by giving a (very) strong negative reward when committing the mistake (here going under a red light) and the agent should learn to do not do this mistake anymore.

It is often better to change actions, rewards or environment rather than acting on hyperparameters in my opinion, but I may be mistaken.

  • $\begingroup$ I agree. If the algorithm works, then its failure in a different environment is probably because of complexity or rewards. $\endgroup$
    – S2673
    May 1, 2021 at 18:51
  • 1
    $\begingroup$ Thanks, I tried normalizing my observations between -1 and 1 and this caused a major improvement. I might also try this for the rewards and actions next. $\endgroup$
    – Philipp
    May 2, 2021 at 16:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .