I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of the environment and rewards are the same. PPO is just a more efficient method compared to TRPO and has proven to be a state-of-the-art method in RL. So, why shouldn't it work? I just thought to ask if someone knows roughly how to transform configuration parameters from TRPO to PPO.

Here some more details about my configurations.


  • Actor loss: $-\log(\pi) * A$ where $A$ are advantages
  • Critic Loss: MSE(predicted_values, discounted return)
  • Desired KL Divergence for Actor and Critic: 0.005
  • Conjugate gradient iterations: 20
  • Residual tolerance in conjugate gradient: 1e-10
  • Damping coefficient for Fisher Product: 1e-3

enter image description here enter image description here enter image description here


  • Actor and Critic optimizer and learning rate: Adam with 0.0001
  • Actor loss: negative minimum of either:
    1. $\frac{\pi}{\pi_{old}} * A$
    2. $clamp(\frac{\pi}{\pi_{old}}, 1-0.1, 1+0.1) * A$
  • Critic loss: MSE(predicted_values, discounted_rewards)
  • Optimization iterations: 10

enter image description here enter image description here enter image description here

The rest of my problem set-up is absolutely the same. But somehow I get completely different results while training, as you can see on the plots above. I also changed learning rates and optimization iterations, gradient clipping, optimizing with mini-batches and $-log(\pi) * A$ as Loss for PPO, but neither helped. Taking importance sampling $\frac{\pi}{\pi_{old}} * A$ as loss for TRPO gives the same results there.

Can someone please help me to understand where could be the problem? Or which parameters I would need to change in PPO?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.