I'm trying to implement the Proximal Policy Optimization (PPO) algorithm (code here), but I am confused about certain concepts.

  1. What is the correct way to implement log probability of a policy (denoted by $\pi_\theta$ below)? $$ L^{C P I}(\theta)=\hat{\mathbb{E}}_{t}\left[\frac{\pi_{\theta}\left(a_{t} | s_{t}\right)}{\pi_{\theta_{\text {old }}}\left(a_{t} | s_{t}\right)} \hat{A}_{t}\right]=\hat{\mathbb{E}}_{t}\left[r_{t}(\theta) \hat{A}_{t}\right] $$ Let's say my old network policy output is oldpolicy_probs=[0.1,0.2,0.6,0.1] and new network policy output is newpolicy_probs=[0.2,0.2,0.4,0.2].

    Do I take the log of this directly, or should I first multiply these with the true label y_true = [0,0,1,0] as implemented here?

  2. ratio = np.mean(np.exp(np.log(newpolicy_probs + 1e-10) - K.log(oldpolicy_probs + 1e-10))*advantage)

    Once I have the ratio and I multiply it with an advantage, why do we take the mean over all actions? I suspect it might be because we are taking estimate $\hat{\mathbb{E}_t}$ but conceptually I don't understand what this gives us. Is my implementation above correct?

  • 1
    $\begingroup$ I have the same question. In this scenario, you have discrete actions right? How were you able to handle this? $\endgroup$ Jun 27 at 18:19
  • $\begingroup$ @AhmedAlagha Please see the implementation of ppo loss here on github. $\endgroup$ Jul 1 at 10:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.