I have built a custom RL environment with gym, which simulates the RL vehicle and potential vehicles in front of the RL vehicle as well as traffic lights (including their state; red, yellow, green). I trained a PPO agent from stable_baselines3 without considering setting of hyperparameters and the agent learned to follow the vehicle in front of it without crashing. However it does not learn to stop at red lights after extensive training.

I tried training it without surrounding vehicles to get more interactions of the RL vehicle with traffic lights and this helped the agent to learn stopping a red light. However when I then continue training of the agent in a new environment with surrounding traffic, the agent again un-learns stopping at red lights.

I am still a novice with RL and do not understand as to why this happens and what I can do here. Should I set hyperparameters? Or try a different model? Or should I exchange the default policy of the PPO model?


The way I dealt with it was by giving a (very) strong negative reward when committing the mistake (here going under a red light) and the agent should learn to do not do this mistake anymore.

It is often better to change actions, rewards or environment rather than acting on hyperparameters in my opinion, but I may be mistaken.

  • $\begingroup$ I agree. If the algorithm works, then its failure in a different environment is probably because of complexity or rewards. $\endgroup$
    – S2673
    May 1 '21 at 18:51
  • 1
    $\begingroup$ Thanks, I tried normalizing my observations between -1 and 1 and this caused a major improvement. I might also try this for the rewards and actions next. $\endgroup$
    – Philipp
    May 2 '21 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.