# Understanding the TensorFlow implementation of the policy gradient method

I was trying to understand the implementation of a basic policy gradient (REINFORCE) method using TensorFlow. I think I got almost everything. The only thing that still bothers me is the loss function implementation.

From the theory, we have that after all the manipulation the gradient of the score function is

$$\nabla_{\theta}J(\theta)=\mathop{\mathbb{E}}\left[\nabla_{\theta}(log(\pi(s,a,\theta)))R(\tau) \right]$$

In this Cartpole example the part relative to the loss function is

    neg_log_prob = tf.nn.softmax_cross_entropy_with_logits_v2(logits = NeuralNetworkOutputs, labels = actions)
loss = tf.reduce_mean(neg_log_prob * discounted_episode_rewards_)


At this point, I do not understand how the definition from above translates into code.

As far as I understood, the functions

tf.nn.softmax_cross_entropy_with_logits_v2(logits = NeuralNetworkOutputs, labels = actions)


returns

log(softmax(NeuralNetworkOutputs))*actions


Which is then multiplied by the discounted returns

log(softmax(NeuralNetworkOutputs))*actions*discounted_episode_rewards_


Within this expression, I do not understand why should we multiply, an expression which looks like the loss function we want, by the value of the action.

• It's not a value of action it's probably one hot vector with 1 on the place where the action you took is placed. You only consider output element of the neural network which represents action that you took. – Brale Apr 30 at 15:37