I'm trying to understand the concept behind the implementation of the OpenAI PPO2 algorithm. The loss function that is minimized is as follows: loss = pg_loss - entropy * ent_coef + vf_loss * vf_coef.

First question: The computation of pg_loss requires to use operations like tf.reduce_mean and tf.maximum. Are these two functions differentiable? Apparently, they are, otherwise, it would not work. Can someone explain why so I can understand the implementation?

Second question: During training, an action is sampled by using the Gumbel Distribution: Noise from such a distribution is added to the logits and then tf.argmax is applied. This index is then used to calculate the negative log-likelihood. However, the tf.argmax should also not be differentiable, so how can this work?

  • $\begingroup$ Please, edit your post to include a link to the implementation you're referring to. $\endgroup$
    – nbro
    Feb 23, 2021 at 10:27


You must log in to answer this question.

Browse other questions tagged .