I'm trying to understand the concept behind the implementation of the OpenAI PPO2 algorithm. The loss function that is minimized is as follows: loss = pg_loss - entropy * ent_coef + vf_loss * vf_coef.
First question: The computation of pg_loss requires to use operations like tf.reduce_mean and tf.maximum. Are these two functions differentiable? Apparently, they are, otherwise, it would not work. Can someone explain why so I can understand the implementation?
Second question: During training, an action is sampled by using the Gumbel Distribution: Noise from such a distribution is added to the logits and then tf.argmax is applied. This index is then used to calculate the negative log-likelihood. However, the tf.argmax should also not be differentiable, so how can this work?
