1
$\begingroup$

I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of the environment and rewards are the same. PPO is just a more efficient method compared to TRPO and has proven to be a state-of-the-art method in RL. So, why shouldn't it work? I just thought to ask if someone knows roughly how to transform configuration parameters from TRPO to PPO.

Here some more details about my configurations.

TRPO

  • Actor loss: $-\log(\pi) * A$ where $A$ are advantages
  • Critic Loss: MSE(predicted_values, discounted return)
  • Desired KL Divergence for Actor and Critic: 0.005
  • Conjugate gradient iterations: 20
  • Residual tolerance in conjugate gradient: 1e-10
  • Damping coefficient for Fisher Product: 1e-3

enter image description here enter image description here enter image description here

PPO

  • Actor and Critic optimizer and learning rate: Adam with 0.0001
  • Actor loss: negative minimum of either:
    1. $\frac{\pi}{\pi_{old}} * A$
    2. $clamp(\frac{\pi}{\pi_{old}}, 1-0.1, 1+0.1) * A$
  • Critic loss: MSE(predicted_values, discounted_rewards)
  • Optimization iterations: 10

enter image description here enter image description here enter image description here


The rest of my problem set-up is absolutely the same. But somehow I get completely different results while training, as you can see on the plots above. I also changed learning rates and optimization iterations, gradient clipping, optimizing with mini-batches and $-log(\pi) * A$ as Loss for PPO, but neither helped. Taking importance sampling $\frac{\pi}{\pi_{old}} * A$ as loss for TRPO gives the same results there.

Can someone please help me to understand where could be the problem? Or which parameters I would need to change in PPO?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.