# Why does PPO lead to a worse performance than TRPO in the same task?

I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of the environment and rewards are the same. PPO is just a more efficient method compared to TRPO and has proven to be a state-of-the-art method in RL. So, why shouldn't it work? I just thought to ask if someone knows roughly how to transform configuration parameters from TRPO to PPO.

Here some more details about my configurations.

### TRPO

• Actor loss: $$-\log(\pi) * A$$ where $$A$$ are advantages
• Critic Loss: MSE(predicted_values, discounted return)
• Desired KL Divergence for Actor and Critic: 0.005
• Residual tolerance in conjugate gradient: 1e-10
• Damping coefficient for Fisher Product: 1e-3

### PPO

• Actor and Critic optimizer and learning rate: Adam with 0.0001
• Actor loss: negative minimum of either:
1. $$\frac{\pi}{\pi_{old}} * A$$
2. $$clamp(\frac{\pi}{\pi_{old}}, 1-0.1, 1+0.1) * A$$
• Critic loss: MSE(predicted_values, discounted_rewards)
• Optimization iterations: 10

The rest of my problem set-up is absolutely the same. But somehow I get completely different results while training, as you can see on the plots above. I also changed learning rates and optimization iterations, gradient clipping, optimizing with mini-batches and $$-log(\pi) * A$$ as Loss for PPO, but neither helped. Taking importance sampling $$\frac{\pi}{\pi_{old}} * A$$ as loss for TRPO gives the same results there.

Can someone please help me to understand where could be the problem? Or which parameters I would need to change in PPO?