I'm trying to understand the concept behind the implementation of the OpenAI PPO2 algorithm. The loss function that is minimized is as follows:
loss = pg_loss - entropy * ent_coef + vf_loss * vf_coef.
First question: The computation of
pg_loss requires to use operations like
tf.maximum. Are these two functions differentiable? Apparently, they are, otherwise, it would not work. Can someone explain why so I can understand the implementation?
Second question: During training, an action is sampled by using the Gumbel Distribution: Noise from such a distribution is added to the
logits and then
tf.argmax is applied. This index is then used to calculate the negative log-likelihood. However, the
tf.argmax should also not be differentiable, so how can this work?