5
$\begingroup$

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients

enter image description here

with the importance sampling term of the policy output probability over the old policy output probability

enter image description here

Could someone please explain this step to me?

I understand once we have done this why we then need to constrain the updates within a 'trust region' (to avoid the πθold increasing the gradient updates outwith the bounds in which the approximations of the gradient direction are accurate), I'm just not sure of the reasons behind including this term in the first place.

$\endgroup$
4
$\begingroup$

I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story:

In the PPO paper, after describing the standard policy gradient objective $L^{PG}$, they mention the following:

While it is appealing to perform multiple steps of optimization on this loss $L^{PG}$ using the same trajectory, doing so is not well-justified, and empirically it often leads to destructively large policy updates

This is because, as soon as you've performed one update using a trajectory generated with the previous policy, you land in an off-policy situation; the experience gained in that trajectory is no longer representative of your current policy, and all the estimators (like the advantage estimator) technically become incorrect.

With importance sampling, you can correct for this. This is also commonly used in multi-step off-policy value learning algorithms. Intuitively, the importance sampling term emphasizes estimates of advantage $\hat{A}_t$ corresponding to actions $a_t$ that have become more likely in the new policy relative to the old policy, and it de-emphasizes advantages corresponding to actions that have already become less likely in the new policy relative to the old policy.

If an action $a_t$ in the old trajectory has already become highly unlikely since that trajectory of experience was generated, we have $\pi_{\theta} (a_t \vert s_t) < \pi_{\theta_{\text{old}}} (a_t \vert s_t)$, which means that $\frac{\pi_{\theta} (a_t \vert s_t)}{\pi_{\theta_{\text{old}}} (a_t \vert s_t)}$ becomes close to $0$, which means that we'll reduce the influence of that particular chunk of experience on our subsequent updates. This makes sense because, due to previous updates since the generation of that trajectory, that particular part of the trajectory has already become highly unlikely anyway, and should therefore no longer be relevant for our updates.

The ability to perform multiple updates using the same (old) trajectory anyway is useful because this increases sample-efficiency, we can re-use the same samples of experience more than once rather than using them once and then discarding them again.

$\endgroup$
0
$\begingroup$

For everybody getting here from google, like me: the log might have been replaced in the loss function, but I think it is still there when taking the gradient of both functions (correct me if I am wrong):
gradient applied

So, the log in the PG function was replaced with r(theta) whose derivate is the same as the log of the PG function (appart from the prportionality factor r(theta)).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.