1
$\begingroup$

I'm trying to implement Proximal Policy Optimization algorithm (code here) but am confused about certain concepts:-

1) What is the correct way to implement log probability of a policy (denoted by pi_theta below)?

enter image description here

Let's say my old network policy output is oldpolicy_probs=[0.1,0.2,0.6,0.1] and new network policy output is newpolicy_probs=[0.2,0.2,0.4,0.2].

Do I take log of this directly, or should I first multiply these with the true label y_true = [0,0,1,0] as implemeted here?


2) ratio = np.mean(np.exp(np.log(newpolicy_probs + 1e-10) - K.log(oldpolicy_probs + 1e-10))*advantage)

Once I have the ratio and I multiply it with advantage, why do we take the mean over all actions? I suspect it might be because we are taking estimate \hat{E_t} but conceptually I don't understand what this gives us. Is my implementation above correct?

Thanks!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.