Pieter Abbeel in his deep rl bootcamp policy gradient lecture derived the gradient of the utility function with respect to $\theta$ as $\nabla U(\theta) \approx \hat{g} = 1/m\sum_{i=1}^m \nabla_\theta logP(\tau^{(i)}; \theta)R(\tau^{(i)})$, where $m$ is the number of rollouts, and $\tau$ represents the trajectory of $s_0,u_0, ..., s_H, u_H$ state action sequences.

He also explains that the gradient increases the log probabilities of trajectories that have positive reward and decreases the log probabilities of trajectories with negative reward, as seen in the picture. From the equation, however, I don't see how the gradient tries to increase the probabilities of the path with positive R?

From the equation, what I understand is that we would want to update $\theta$ in a way that moves in the direction of $\nabla U(\theta)$ so that the overall utility is maximised, and this entails computing the gradient log probability of a trajectory.

Also, why is $\theta$ omitted in $R(\tau^{(i)})$, since $\tau$ depends on the policy which is dependent on $\theta$ ?

enter image description here


Think of a surface where z-axis is $U$, x and y axis are $\theta_{1}$ and $\theta_{2}$ accordingly. Since you are following the gradient direction with respect to $\theta$ vector, that means you are moving at the direction that increases the $U$. If $R(\tau)$ is positive, you are moving towards the uphill direction and vice-versa. More formally you will say the following:

In the policy gradient algorithm, our update step is:

$ \theta_{new} = \theta_{old} + \alpha \nabla_{\theta}U(\theta) $

So, if we select a very bad trajectory the sum of all rewards ($R(\tau)$) will be negative and the following update will shift the $\theta_{new}$ vector away from $\nabla_{\theta}U(\theta)$ vector.

If we select a good trajectory means $R(\tau)$ is positive, the update will shift $\theta_{new}$ vector towards the $\nabla_{\theta}U(\theta)$ vector. So, it will increase the probability of selecting path with positive $R$

  • $\begingroup$ Hey thanks for your answer. The reward, $R(\tau)$ is a scalar quantity and the grad log probability of a trajectory is negative and the grad log probability of the trajectory returns the gradient of the weights in the neural network rite ? $\endgroup$ – calveeen May 12 '20 at 10:59
  • $\begingroup$ yes, $R(\tau)$ is a scalar quantity, but the later part you say can not able to understand what you mean by that. $\endgroup$ – Swakshar Deb May 12 '20 at 11:02
  • $\begingroup$ the grad log probability of a trajectory i meant refers to the term $\nabla_{\theta}log P(\tau^{(i)}, \theta)$. so the scalar quantity of the reward is multiplied by this vector to give the gradient of the Utility function w.r.t $\theta$ rite ? $\endgroup$ – calveeen May 12 '20 at 11:05
  • $\begingroup$ yes. reward$(R(\tau)$ is scalar and you are multiplying the scalar$(R)$ with the vector $\nabla_{\theta}logP(\tau^{i},\theta)$. $\endgroup$ – Swakshar Deb May 12 '20 at 11:10
  • $\begingroup$ how would a negative reward shift $\theta_{new}$ to not favour that bad reward trajectory ? because whether or not a positive scalar or negative scalar is multiplied by the weights, $\theta$ will be updated to move in the direction of the gradient, which means $\theta$ the new $\theta$ will overall increase the utility received but the equation does not show how it affects the log probability of a good or bad trajectory ? $\endgroup$ – calveeen May 12 '20 at 11:11

The grad log probability of the trajectory parameterised by $\theta$ tells us the direction $\theta$ should move to increase the probability of that trajectory $P(\tau;\theta)$ the most.

If the reward is positive, $\nabla U(\theta)$ tells us how much we want to increase/decrease the probability of that path $\tau$. The scalar quantity $R(\tau)$ determines the magnitude and direction of shift. If $R(\tau)$ is positive, and $\theta$ is updated based on equation $\theta_{new}$ = $\theta_{old} + \alpha\nabla_{\theta}U(\theta)$, then $\theta$ will move in the direction of it's steepest increase, leading to an increase in probability of $\tau$. If $R(\tau)$ is negative, then $\theta$ moves in the direction of steepest decrease, leading to a decrease in probability of $\tau$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.