When trying to implement my own PPO (Proximal Policy Optimizer), I came accross two different implementations :

  • Exploration with action std :
    1. Collect trajectories on N timesteps, by using policy-centered distribution with progressively-trained std variable for exploration
    2. Train policy function on K steps
    3. Train value function on K steps

Ex: The openAI implementation

  • Exploration with entropy :
    1. Collect trajectories on N timesteps, by using policy function directly
    2. Train policy and value function at the same time on K steps, with common loss for the two model, with additional entropy bonus for exploration purpose.

Ex: PPO algorithm, as described in the official paper

I would like to know what are the pros / cons between this two algorithms ?

Is this specific to PPO, or is this a classic question concerning policy gradients algorithms, in general ?

Could not find much relevant documentation about it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.