How does PPO account for the last reward?

Question

I was implementing PPO for the lunar lander environment in openai gym, but my agent seems to be getting stuck at a score of ~-80. On the website it says the agent gets rewarded +100 or -100 if it comes to a rest or crashes at the end. However, my time horizon is set to 20, which leads me to think the last reward is not properly accounted for (since episode lengths arent always a multiple of 20) and thus, the agent predicts bogous values for future states which causes the poor results. This begs the question: how does ppo account for the final reward? Do we just hope the algorithm stumbles upon an episode where the length is a multiple of the time horizon, or is it that the environment is programmed to wait a few steps after finishing to ensure that the final reward gets accounted for?

Answer 1

I actually feel kinda dumb writing this, but the update is made if the TOTAL environment steps is a multiple of the time horizon. I did, recode my PPO algorithm, and it seems to eventually solve the lunar lander environment, albeit very slowly (taking around 650k training steps from 1700 episodes to solve), and the training is very unstable, the agent might average around 100 points and drop to an avg of -80 in the next 500 episodes and suddenly jump back to 200 points (my lr is 3e-5).