I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of the environment and rewards are the same. PPO is just a more efficient method compared to TRPO and has proven to be a state-of-the-art method in RL. So, why shouldn't it work? I just thought to ask if someone knows roughly how to transform configuration parameters from TRPO to PPO.
Here some more details about my configurations.
TRPO
- Actor loss: $-\log(\pi) * A$ where $A$ are advantages
- Critic Loss: MSE(predicted_values, discounted return)
- Desired KL Divergence for Actor and Critic: 0.005
- Conjugate gradient iterations: 20
- Residual tolerance in conjugate gradient: 1e-10
- Damping coefficient for Fisher Product: 1e-3
PPO
- Actor and Critic optimizer and learning rate: Adam with 0.0001
- Actor loss: negative minimum of either:
- $\frac{\pi}{\pi_{old}} * A$
- $clamp(\frac{\pi}{\pi_{old}}, 1-0.1, 1+0.1) * A$
- Critic loss: MSE(predicted_values, discounted_rewards)
- Optimization iterations: 10
The rest of my problem set-up is absolutely the same. But somehow I get completely different results while training, as you can see on the plots above. I also changed learning rates and optimization iterations, gradient clipping, optimizing with mini-batches and $-log(\pi) * A$ as Loss for PPO, but neither helped. Taking importance sampling $\frac{\pi}{\pi_{old}} * A$ as loss for TRPO gives the same results there.
Can someone please help me to understand where could be the problem? Or which parameters I would need to change in PPO?
