I'm trying to implement Proximal Policy Optimization algorithm (code here) but am confused about certain concepts:-

1) What is the correct way to implement log probability of a policy (denoted by pi_theta below)?

enter image description here

Let's say my old network policy output is oldpolicy_probs=[0.1,0.2,0.6,0.1] and new network policy output is newpolicy_probs=[0.2,0.2,0.4,0.2].

Do I take log of this directly, or should I first multiply these with the true label y_true = [0,0,1,0] as implemeted here?

2) ratio = np.mean(np.exp(np.log(newpolicy_probs + 1e-10) - K.log(oldpolicy_probs + 1e-10))*advantage)

Once I have the ratio and I multiply it with advantage, why do we take the mean over all actions? I suspect it might be because we are taking estimate \hat{E_t} but conceptually I don't understand what this gives us. Is my implementation above correct?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.