I was trying to understand the implementation of a basic policy gradient (REINFORCE) method using TensorFlow. I think I got almost everything. The only thing that still bothers me is the loss function implementation.
From the theory, we have that after all the manipulation the gradient of the score function is
$$\nabla_{\theta}J(\theta)=\mathop{\mathbb{E}}\left[\nabla_{\theta}(log(\pi(s,a,\theta)))R(\tau) \right]$$
In this Cartpole example the part relative to the loss function is
neg_log_prob = tf.nn.softmax_cross_entropy_with_logits_v2(logits = NeuralNetworkOutputs, labels = actions)
loss = tf.reduce_mean(neg_log_prob * discounted_episode_rewards_)
At this point, I do not understand how the definition from above translates into code.
As far as I understood, the functions
tf.nn.softmax_cross_entropy_with_logits_v2(logits = NeuralNetworkOutputs, labels = actions)
returns
log(softmax(NeuralNetworkOutputs))*actions
Which is then multiplied by the discounted returns
log(softmax(NeuralNetworkOutputs))*actions*discounted_episode_rewards_
Within this expression, I do not understand why should we multiply, an expression which looks like the loss function we want, by the value of the action.