1
$\begingroup$

When trying to implement my own PPO (Proximal Policy Optimizer), I came accross two different implementations :

  • Exploration with action std :
    1. Collect trajectories on N timesteps, by using policy-centered distribution with progressively-trained std variable for exploration
    2. Train policy function on K steps
    3. Train value function on K steps

Ex: The openAI implementation

  • Exploration with entropy :
    1. Collect trajectories on N timesteps, by using policy function directly
    2. Train policy and value function at the same time on K steps, with common loss for the two model, with additional entropy bonus for exploration purpose.

Ex: PPO algorithm, as described in the official paper

I would like to know what are the pros / cons between this two algorithms ?

Is this specific to PPO, or is this a classic question concerning policy gradients algorithms, in general ?

Could not find much relevant documentation about it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.