In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients

$$L^{PG}(\theta) = \hat{\mathbb{E}}_t[\log \pi_{\theta}(a_t | s_t)\hat{A}_t]$$

with the importance sampling term of the policy output probability over the old policy output probability

$$L^{IS}_{\theta_{old}}(\theta) = \hat{\mathbb{E}}_t \left[\frac{\pi_{\theta}(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}\hat{A}_t \right]$$

Could someone please explain this step to me?

I understand once we have done this why we then need to constrain the updates within a 'trust region' (to avoid the $\pi_{\theta_{old}}$ increasing the gradient updates outwith the bounds in which the approximations of the gradient direction are accurate), I'm just not sure of the reasons behind including this term in the first place.

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients

$$L^{PG}(\theta) = \hat{\mathbb{E}}_t[\log \pi_{\theta}(a_t | s_t)\hat{A}_t],$$

with the importance sampling term of the policy output probability over the old policy output probability

$$L^{IS}_{\theta_{old}}(\theta) = \hat{\mathbb{E}}_t \left[\frac{\pi_{\theta}(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}\hat{A}_t \right]$$

Could someone please explain this step to me?

I understand once we have done this why we then need to constrain the updates within a 'trust region' (to avoid the $\pi_{\theta_{old}}$ increasing the gradient updates outwith the bounds in which the approximations of the gradient direction are accurate). I'm just not sure of the reasons behind including this term in the first place.

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients

$L^{PG}(\theta) = \hat{\mathbb{E}}_t[\log \pi_{\theta}(a_t | s_t)\hat{A}_t]$

with the importance sampling term of the policy output probability over the old policy output probability

$L^{IS}_{\theta_{old}}(\theta) = \hat{\mathbb{E}}_t[\frac{\pi_{\theta}(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}\hat{A}_t]$

Could someone please explain this step to me?

I understand once we have done this why we then need to constrain the updates within a 'trust region' (to avoid the π_θold increasing the gradient updates outwith the bounds in which the approximations of the gradient direction are accurate), I'm just not sure of the reasons behind including this term in the first place.

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients

$$L^{PG}(\theta) = \hat{\mathbb{E}}_t[\log \pi_{\theta}(a_t | s_t)\hat{A}_t]$$

with the importance sampling term of the policy output probability over the old policy output probability

$$L^{IS}_{\theta_{old}}(\theta) = \hat{\mathbb{E}}_t \left[\frac{\pi_{\theta}(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}\hat{A}_t \right]$$

Could someone please explain this step to me?

I understand once we have done this why we then need to constrain the updates within a 'trust region' (to avoid the $\pi_{\theta_{old}}$ increasing the gradient updates outwith the bounds in which the approximations of the gradient direction are accurate), I'm just not sure of the reasons behind including this term in the first place.

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients

with the importance sampling term of the policy output probability over the old policy output probability

Could someone please explain this step to me?

I understand once we have done this why we then need to constrain the updates within a 'trust region' (to avoid the π_θold increasing the gradient updates outwith the bounds in which the approximations of the gradient direction are accurate), I'm just not sure of the reasons behind including this term in the first place.

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients

$L^{PG}(\theta) = \hat{\mathbb{E}}_t[\log \pi_{\theta}(a_t | s_t)\hat{A}_t]$

with the importance sampling term of the policy output probability over the old policy output probability

$L^{IS}_{\theta_{old}}(\theta) = \hat{\mathbb{E}}_t[\frac{\pi_{\theta}(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}\hat{A}_t]$

Could someone please explain this step to me?

I understand once we have done this why we then need to constrain the updates within a 'trust region' (to avoid the π_θold increasing the gradient updates outwith the bounds in which the approximations of the gradient direction are accurate), I'm just not sure of the reasons behind including this term in the first place.