I was watching a lecture on Policy Gradient and had difficulty following it when importance sampling was introduced.

It was shown that the gradient of the objective can be written as $$\nabla_\theta U(\theta) = E_{\tau \sim \pi_\theta}[\nabla_\theta P(\tau; \theta) R(\tau)],$$ where $P(\tau)$ denotes the probability of getting the trajectory $\tau$ and reward $R(\tau) = \sum_i R(\tau^{(i)})$ is the sum of the rewards for trajectory.

Then comes the part of my confusion. It was demonstrated that the same result can be derived with importance sampling and have no idea what this bit means (https://youtu.be/S_gwYj1Q-44?t=1404 and onwards until the next slide, takes around a minute). It's said:

If we're curious about other policies, even if we have trajectories from an old policy, we can understand how good this other policy is going to be, let's look locally if we deviate from $\theta_{old}$, and it gives us the same gradient

But I don't see how:

  1. The gradient of new policy evaluated at $\theta_{old}$ is the same as the gradient (I think he means in the same form as the derivation without importance sampling) at the location $\theta$.

  2. We even can evaluate the gradient of a policy at a different location other than $\theta_{old}$, which we currently have. We can find the derivatives at out current location, but we cannot do that if we don't have analytical expression of the function. He mentions finite perturbation, which I think achieves this, but I don't see how. To this, he also says:

Also, you can do more, if we use the original derivation of the gradient, we can change the gradient and then use this equation to estimate how good the update was rather than just blindly trusting your gradient put you in the good spot.

In general, I'd like to get a better picture of the benefits of formulating the problem in this way

  • $\begingroup$ I don't think I can currently answer this question, but, to clarify something, do you have an idea of what importance sampling is or can be used for? $\endgroup$
    – nbro
    Jan 23 at 15:08
  • $\begingroup$ @nbro I think I do, the benefit is that it allows us to use data that were gathered from another policy by correcting the computation so to make it "seem" like the data came from our new policy $\endgroup$ Jan 23 at 19:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.