I was watching a lecture on Policy Gradient and had difficulty following it when importance sampling was introduced.
It was shown that the gradient of the objective can be written as $$\nabla_\theta U(\theta) = E_{\tau \sim \pi_\theta}[\nabla_\theta P(\tau; \theta) R(\tau)],$$ where $P(\tau)$ denotes the probability of getting the trajectory $\tau$ and reward $R(\tau) = \sum_i R(\tau^{(i)})$ is the sum of the rewards for trajectory.
Then comes the part of my confusion. It was demonstrated that the same result can be derived with importance sampling and have no idea what this bit means (https://youtu.be/S_gwYj1Q-44?t=1404 and onwards until the next slide, takes around a minute). It's said:
If we're curious about other policies, even if we have trajectories from an old policy, we can understand how good this other policy is going to be, let's look locally if we deviate from $\theta_{old}$, and it gives us the same gradient
But I don't see how:
The gradient of new policy evaluated at $\theta_{old}$ is the same as the gradient (I think he means in the same form as the derivation without importance sampling) at the location $\theta$.
We even can evaluate the gradient of a policy at a different location other than $\theta_{old}$, which we currently have. We can find the derivatives at out current location, but we cannot do that if we don't have analytical expression of the function. He mentions finite perturbation, which I think achieves this, but I don't see how. To this, he also says:
Also, you can do more, if we use the original derivation of the gradient, we can change the gradient and then use this equation to estimate how good the update was rather than just blindly trusting your gradient put you in the good spot.
In general, I'd like to get a better picture of the benefits of formulating the problem in this way
