3
$\begingroup$

I'm trying to implement the Proximal Policy Optimization (PPO) algorithm (code here), but I am confused about certain concepts.

  1. What is the correct way to implement log probability of a policy (denoted by $\pi_\theta$ below)? $$ L^{C P I}(\theta)=\hat{\mathbb{E}}_{t}\left[\frac{\pi_{\theta}\left(a_{t} | s_{t}\right)}{\pi_{\theta_{\text {old }}}\left(a_{t} | s_{t}\right)} \hat{A}_{t}\right]=\hat{\mathbb{E}}_{t}\left[r_{t}(\theta) \hat{A}_{t}\right] $$ Let's say my old network policy output is oldpolicy_probs=[0.1,0.2,0.6,0.1] and new network policy output is newpolicy_probs=[0.2,0.2,0.4,0.2].

    Do I take the log of this directly, or should I first multiply these with the true label y_true = [0,0,1,0] as implemented here?

  2. ratio = np.mean(np.exp(np.log(newpolicy_probs + 1e-10) - K.log(oldpolicy_probs + 1e-10))*advantage)

    Once I have the ratio and I multiply it with an advantage, why do we take the mean over all actions? I suspect it might be because we are taking estimate $\hat{\mathbb{E}_t}$ but conceptually I don't understand what this gives us. Is my implementation above correct?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ I have the same question. In this scenario, you have discrete actions right? How were you able to handle this? $\endgroup$
    – Ahmed Alagha
    Jun 27, 2021 at 18:19
  • $\begingroup$ @AhmedAlagha Please see the implementation of ppo loss here on github. $\endgroup$
    – Chintan Trivedi
    Jul 1, 2021 at 10:06

0

Reset to default

You must log in to answer this question.