I have implemented the total loss of my PPO objective as follows:-
total_loss = critic_discount * critic_loss + actor_loss - entropy_beta * K.mean(-(newpolicy_probs * K.log(newpolicy_probs)))
After training for a few epochs, the entropy term becomes "nan" for some reason. I used
tf.Print() to see the new policy probabilities when the entropy becomes undefined, it is as follows-
new policy probs: [[6.1029973e-06 1.93471514e-08 0.000299338106...]...]
I am not clear as to why taking log of these small probabilities is coming out as
nan. Any idea how to prevent this?