# Policy gradient loss for neural network training

Say i want to train a neural network with 10 classes as outputs and use categorical_cross_entropy as a loss function in keras. This will try to fit the training data as best as possible irregardless of the outcome (i.e. value). If I want to take value into account, I have to use something like a policy gradient RL algorithm. How do I formulate the loss of policy gradient algorithm in this case ?

The standard categorical cross entropy loss function is as follows where y_ = true value, and y = predicted value:

   loss = -mean( y_ * log(y))


I am thinking to just multiply the true value by the reward and still use the categorical cross entropy of keras i.e.

   y_ = y_ * reward
loss = -mean( y_* log(y) )


Is my interpretation correct ?

• Do you actually have a RL problem to solve? It is not clear whether you are trying to train a policy network to solve an environment, or trying to apply concepts from reinforcement learning on something else. If you could edit your question to include some context about the problem you are trying to solve, it may clarify what approach would be best. – Neil Slater Apr 20 at 20:54
• I am using this for playing a game using combined policy+value neural network just like alphago zero. For supervised training from human games, cross entropy loss is sufficient. But for self-play training I want to use the policy gradient algorithm as described in my post, but I am not sure how to compute the loss function for it. – danny Apr 20 at 21:01
• OK that makes sense. What are you intending to use for the "true value" in the context of your question? Are you using MCTS search to establish a refined policy during self-play (like AlphaZero)? Have you read and are trying to replicate the AlphaZero paper? I think that the loss functions are included in that paper . . . at least I have read a few replications which appear to quote the loss functions. – Neil Slater Apr 20 at 21:03
• The 800 playouts mcts training is too expensive for me, so i am thinking to try out a 1 playout training instead -- i.e. use policy gradient method. They actually did that in their first paper for RL training of the policy network. I think with 800 playouts mcts, fitting the policy network even to the loosing player's moves is Ok to do. – danny Apr 20 at 21:09
• "I think with 800 playouts mcts, fitting the policy network even to the loosing player's moves is Ok to do" Yes I think that is correct - the MCTS search is directly improving the policy, and one of the loss functions is loss between MCTS-discovered improved policy and existing policy. I'm not 100% clear on what you are doing - are you intending to run any look-ahead planning in your game, or run something more like Actor-Critic where the agent learns by doing without planning (this should still be OK and does have a loss function - in fact a few variations) – Neil Slater Apr 20 at 21:14