3
$\begingroup$

I was trying to understand the implementation of a basic policy gradient (REINFORCE) method using TensorFlow. I think I got almost everything. The only thing that still bothers me is the loss function implementation.

From the theory, we have that after all the manipulation the gradient of the score function is

$$\nabla_{\theta}J(\theta)=\mathop{\mathbb{E}}\left[\nabla_{\theta}(log(\pi(s,a,\theta)))R(\tau) \right]$$

In this Cartpole example the part relative to the loss function is

    neg_log_prob = tf.nn.softmax_cross_entropy_with_logits_v2(logits = NeuralNetworkOutputs, labels = actions)
    loss = tf.reduce_mean(neg_log_prob * discounted_episode_rewards_) 

At this point, I do not understand how the definition from above translates into code.

As far as I understood, the functions

tf.nn.softmax_cross_entropy_with_logits_v2(logits = NeuralNetworkOutputs, labels = actions)

returns

log(softmax(NeuralNetworkOutputs))*actions

Which is then multiplied by the discounted returns

log(softmax(NeuralNetworkOutputs))*actions*discounted_episode_rewards_

Within this expression, I do not understand why should we multiply, an expression which looks like the loss function we want, by the value of the action.

$\endgroup$
  • $\begingroup$ It's not a value of action it's probably one hot vector with 1 on the place where the action you took is placed. You only consider output element of the neural network which represents action that you took. $\endgroup$ – Brale Apr 30 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.