Copy from my reddit post: (Sorry if this does not fit here, please tell me and i delete it) Help regarding I'm working on an implementation of PPO, which i plan to use in my (Bachelors) Thesis. To test whether my implementation works, i want to use the LunarLanderContinuous-v2 Environment. Now my implementation seems to work just fine, but plateues much too early - At an average reward of ~ -1.8 reward per timestep, where the goal should be somewhere around ~ +2,5 reward per timestep. As the implementation generally learns i am somewhat confused, as to why it then pleateus so early. Some details regarding my implementation, also here is the github repo:

  • I use parallelized environments via openai's subproc_vecenv
  • I use the Actor Critic Version of PPO
  • I use Generalized Advantage Estimation as my Advantage term
  • I only use finished runs (every run used in training has reached a terminal state)
  • Even though Critic loss in the graphic below looks small it is actually rather large, as the rewards are normalized and therefor the value targets are actually rather small

  • The Critic seemingly predicts a value independent of the state it is fed - that is it predicts for every state just the average over all the values. That seems like harsh underfitting, which is weird as the network is already rather large for the problem in my opinion. But this seems to be the most likely cause for the problem in my opinion.Training Progress Edit1: Added image

  • $\begingroup$ Interesting question. Did you succeed? Is the code available somewhere? I have similar issues... $\endgroup$
    – sqp_125
    Mar 28 '20 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.