Why is the log probability replaced with the importance sampling in the loss function?

Question

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients

$$L^{PG}(\theta) = \hat{\mathbb{E}}_t[\log \pi_{\theta}(a_t | s_t)\hat{A}_t],$$

with the importance sampling term of the policy output probability over the old policy output probability

$$L^{IS}_{\theta_{old}}(\theta) = \hat{\mathbb{E}}_t \left[\frac{\pi_{\theta}(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}\hat{A}_t \right]$$

Could someone please explain this step to me?

I understand once we have done this why we then need to constrain the updates within a 'trust region' (to avoid the $\pi_{\theta_{old}}$ increasing the gradient updates outwith the bounds in which the approximations of the gradient direction are accurate). I'm just not sure of the reasons behind including this term in the first place.

Answer 1

I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story:

In the PPO paper, after describing the standard policy gradient objective $L^{PG}$, they mention the following:

While it is appealing to perform multiple steps of optimization on this loss $L^{PG}$ using the same trajectory, doing so is not well-justified, and empirically it often leads to destructively large policy updates

This is because, as soon as you've performed one update using a trajectory generated with the previous policy, you land in an off-policy situation; the experience gained in that trajectory is no longer representative of your current policy, and all the estimators (like the advantage estimator) technically become incorrect.

With importance sampling, you can correct for this. This is also commonly used in multi-step off-policy value learning algorithms. Intuitively, the importance sampling term emphasizes estimates of advantage $\hat{A}_t$ corresponding to actions $a_t$ that have become more likely in the new policy relative to the old policy, and it de-emphasizes advantages corresponding to actions that have already become less likely in the new policy relative to the old policy.

If an action $a_t$ in the old trajectory has already become highly unlikely since that trajectory of experience was generated, we have $\pi_{\theta} (a_t \vert s_t) < \pi_{\theta_{\text{old}}} (a_t \vert s_t)$, which means that $\frac{\pi_{\theta} (a_t \vert s_t)}{\pi_{\theta_{\text{old}}} (a_t \vert s_t)}$ becomes close to $0$, which means that we'll reduce the influence of that particular chunk of experience on our subsequent updates. This makes sense because, due to previous updates since the generation of that trajectory, that particular part of the trajectory has already become highly unlikely anyway, and should therefore no longer be relevant for our updates.

The ability to perform multiple updates using the same (old) trajectory anyway is useful because this increases sample-efficiency, we can re-use the same samples of experience more than once rather than using them once and then discarding them again.

Answer 2

For everybody getting here from google, like me: the $\log$ might have been replaced in the loss function, but I think it is still there when taking the gradient of both functions (correct me, if I am wrong):

$$\begin{aligned} \nabla_{\theta} L^{P G}(\theta) &=\nabla_{\theta} \hat{E}_{t}\left[\log \pi_{\theta}\left(a_{t} \mid s_{t}\right) \hat{A}_{t}\right] \\ &=\hat{E}_{t}\left[\nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) \hat{A}_{t}\right] \end{aligned}$$

and

$$ \begin{aligned} \nabla_{\theta} L^{I S}(\theta)=& \nabla_{\theta} \hat{E}_{t} \left[\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} \hat{A}_{t}\right] \\ &=\hat{E}_{t} \left[\nabla_{\theta} \frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} \hat{A}_{t}\right] \\ &=\hat{E}_{t} \left[\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} \frac{\nabla_{\theta} \pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta}\left(a_{t} \mid s_{t}\right)} \hat{A}_{t}\right] \\ &=\hat{E}_{t}\left[\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) \hat{A}_{t}\right] \end{aligned} $$

So, the $\pi_{\theta}\left(a_{t} \mid s_{t}\right) $ in the PG function was replaced with $\frac{\pi_{\theta}\left(a_{t} \mid s_{t}\right)}{\pi_{\theta_{\text {old}}}\left(a_{t} \mid s_{t}\right)} $ whose derivate is the same as the log of the PG function (apart from the proportionality factor).