How does the gradient increase the probabilities of the path with a positive reward in policy gradient?

Question

Pieter Abbeel in his deep rl bootcamp policy gradient lecture derived the gradient of the utility function with respect to $\theta$ as $\nabla U(\theta) \approx \hat{g} = 1/m\sum_{i=1}^m \nabla_\theta logP(\tau^{(i)}; \theta)R(\tau^{(i)})$, where $m$ is the number of rollouts, and $\tau$ represents the trajectory of $s_0,u_0, ..., s_H, u_H$ state action sequences.

He also explains that the gradient increases the log probabilities of trajectories that have positive reward and decreases the log probabilities of trajectories with negative reward, as seen in the picture. From the equation, however, I don't see how the gradient tries to increase the probabilities of the path with positive R?

From the equation, what I understand is that we would want to update $\theta$ in a way that moves in the direction of $\nabla U(\theta)$ so that the overall utility is maximised, and this entails computing the gradient log probability of a trajectory.

Also, why is $\theta$ omitted in $R(\tau^{(i)})$, since $\tau$ depends on the policy which is dependent on $\theta$ ?

Answer 1

Think of a surface where z-axis is $U$, x and y axis are $\theta_{1}$ and $\theta_{2}$ accordingly. Since you are following the gradient direction with respect to $\theta$ vector, that means you are moving at the direction that increases the $U$. If $R(\tau)$ is positive, you are moving towards the uphill direction and vice-versa. More formally you will say the following:

In the policy gradient algorithm, our update step is:

$ \theta_{new} = \theta_{old} + \alpha \nabla_{\theta}U(\theta) $

So, if we select a very bad trajectory the sum of all rewards ($R(\tau)$) will be negative and the following update will shift the $\theta_{new}$ vector away from $\nabla_{\theta}U(\theta)$ vector.

If we select a good trajectory means $R(\tau)$ is positive, the update will shift $\theta_{new}$ vector towards the $\nabla_{\theta}U(\theta)$ vector. So, it will increase the probability of selecting path with positive $R$

Answer 2

The grad log probability of the trajectory parameterised by $\theta$ tells us the direction $\theta$ should move to increase the probability of that trajectory $P(\tau;\theta)$ the most.

If the reward is positive, $\nabla U(\theta)$ tells us how much we want to increase/decrease the probability of that path $\tau$. The scalar quantity $R(\tau)$ determines the magnitude and direction of shift. If $R(\tau)$ is positive, and $\theta$ is updated based on equation $\theta_{new}$ = $\theta_{old} + \alpha\nabla_{\theta}U(\theta)$, then $\theta$ will move in the direction of it's steepest increase, leading to an increase in probability of $\tau$. If $R(\tau)$ is negative, then $\theta$ moves in the direction of steepest decrease, leading to a decrease in probability of $\tau$.