I am taking Berkeley’s CS285 via self-study. On this particular lecture regarding Policy Gradient, I am very confused about the inconsistency between the concept explanation and the demonstration of code snippet. I am new to RL and hope someone could clarify this for me.


1.The lecture defines policy gradient as follow:

enter image description here

log(pi_theta(a | s)) denotes the log probability of action given state under policy parameterized by theta

gradient log(pi_theta(a |s)) denotes the gradient of parameter theta with respect to the predicted log probability of action

2.The lecture defines a pseudo-loss.By auto differentiate the pseudo-loss, we recovery the policy gradient.

enter image description here

Here Q_hat is short hand of sum of r(s_i_t, a_i_t) in the Policy gradient equation under 1)

  1. The lecture then proceeds to gives a pseudo-code implementation of 2)

enter image description here

My confusion

From 1) above,

gradient log(pi_theta(a |s)) denotes the gradient of parameter theta with respect to the predicted log probability of action , not a loss value calculated from a label action and predicted action.

Why does the below in 2) implies that gradient log(pi_theta(a |s)) just morph into output of loss function instead of just predicted action probability as defined in 1) ?

enter image description here


In this pseudo-code implementation,

enter image description here

Particularly, this line below.

negative_likelihoods = tf.nn.softmax_cross_entrophy_with_logis(labels=actions, logits=logits)

Where does the actions even coming from ? If it comes from collected trajectory, aren’t the actions result of logits = policy.predictions(states) to begin with ? Then won’t tf.nn.softmax_cross_entrophy_with_logis(labels=actions, logits=logits) always return 0 ?

  1. Based on the definition of policy gradient in 1), shouldn’t the implementation of pseudo-loss be like below ?

enter image description here

# Given:
# actions - (N*T) x Da tensor of actions
# states - (N*T) x Ds tensor of states
# q_values – (N*T) x 1 tensor of estimated state-action values
# Build the graph:
logits = policy.predictions(states) # This should return (N*T) x Da tensor of action logits 
weighted_predicted_probability = tf.multiply(torch.softmax(logits), q_values)
loss = tf.reduce_mean(weighted_predicted_probability )
gradients = loss.gradients(loss, variables)

  • 1
    $\begingroup$ Hello. Welcome to Artificial Intelligence Stack Exchange. I would suggest that, rather than saying in the title that you are confused, you should put in the title your main specific question that you have. An example of a question is "Why is this equation X inconsistent with this implementation?" or something like that. That's what I mean by question. $\endgroup$ – nbro Feb 11 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.