0
$\begingroup$

I have implemented the total loss of my PPO objective as follows:-

total_loss = critic_discount * critic_loss + actor_loss -  entropy_beta * K.mean(-(newpolicy_probs * K.log(newpolicy_probs)))

After training for a few epochs, the entropy term becomes "nan" for some reason. I used tf.Print() to see the new policy probabilities when the entropy becomes undefined, it is as follows-

new policy probs: [[6.1029973e-06 1.93471514e-08 0.000299338106...]...]

I am not clear as to why taking log of these small probabilities is coming out as nan. Any idea how to prevent this?

$\endgroup$
  • 1
    $\begingroup$ Log of zero is mathematically undefined. On a computer, the log is likely approximated (it depends on the implementation). Furthermore, the multiplication of small numbers creates even smaller numbers (e.g. the product of $0.1*0.1 = 0.01$), which leads quite quickly to the "vanishing problem". Are you using floats (single-precision) or doubles? If you're using floats, maybe try using doubles? Or maybe try to manually handle the case where too small probabilities that are passed to the log. If this answers your question, let me know, and I will convert this comment to an answer. $\endgroup$ – nbro Jul 24 at 20:18
  • $\begingroup$ However, note that this question is off-topic here. You should ask it on Stack Overflow, given that this is a problem related to the implementation. Here we focus more on conceptual and philosophical questions. $\endgroup$ – nbro Jul 24 at 20:21
  • $\begingroup$ I addressed my implemetation issue with the solution in my answer below. However, I still have conceptual doubts so I opened a separate question here: ai.stackexchange.com/questions/13577/… , thanks for the advice! $\endgroup$ – Chintan Trivedi Jul 25 at 14:14
  • $\begingroup$ I did handle small values manually by adding 1e-10 so if you want to post it as an answer I'll accept it. $\endgroup$ – Chintan Trivedi Jul 25 at 14:17
1
$\begingroup$

I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.