I'm trying to understand the concept behind the implementation of the OpenAI PPO2 algorithm. The loss function that is minimized is as follows: loss = pg_loss - entropy * ent_coef + vf_loss * vf_coef
.
First question: The computation of pg_loss
requires to use operations like tf.reduce_mean
and tf.maximum
. Are these two functions differentiable? Apparently, they are, otherwise, it would not work. Can someone explain why so I can understand the implementation?
Second question: During training, an action is sampled by using the Gumbel Distribution: Noise from such a distribution is added to the logits
and then tf.argmax
is applied. This index is then used to calculate the negative log-likelihood. However, the tf.argmax
should also not be differentiable, so how can this work?