# Understanding log probabilities of actions in the PPO objective

I'm trying to implement the Proximal Policy Optimization (PPO) algorithm (code here), but I am confused about certain concepts.

1. What is the correct way to implement log probability of a policy (denoted by $$\pi_\theta$$ below)? $$L^{C P I}(\theta)=\hat{\mathbb{E}}_{t}\left[\frac{\pi_{\theta}\left(a_{t} | s_{t}\right)}{\pi_{\theta_{\text {old }}}\left(a_{t} | s_{t}\right)} \hat{A}_{t}\right]=\hat{\mathbb{E}}_{t}\left[r_{t}(\theta) \hat{A}_{t}\right]$$ Let's say my old network policy output is oldpolicy_probs=[0.1,0.2,0.6,0.1] and new network policy output is newpolicy_probs=[0.2,0.2,0.4,0.2].

Do I take the log of this directly, or should I first multiply these with the true label y_true = [0,0,1,0] as implemented here?

2. ratio = np.mean(np.exp(np.log(newpolicy_probs + 1e-10) - K.log(oldpolicy_probs + 1e-10))*advantage)

Once I have the ratio and I multiply it with an advantage, why do we take the mean over all actions? I suspect it might be because we are taking estimate $$\hat{\mathbb{E}_t}$$ but conceptually I don't understand what this gives us. Is my implementation above correct?

• I have the same question. In this scenario, you have discrete actions right? How were you able to handle this? Jun 27 at 18:19
• @AhmedAlagha Please see the implementation of ppo loss here on github. Jul 1 at 10:06