Skip to main content
2 of 5
edited tags
nbro
  • 41.4k
  • 12
  • 114
  • 205

TRPO/PPO importance sampling term in loss function

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients

enter image description here

with the importance sampling term of the policy output probability over the old policy output probability

enter image description here

Could someone please explain this step to me?

I understand once we have done this why we then need to constrain the updates within a 'trust region' (to avoid the πθold increasing the gradient updates outwith the bounds in which the approximations of the gradient direction are accurate), I'm just not sure of the reasons behind including this term in the first place.

Mark
  • 106
  • 4