# Which off-policy policy gradient estimator has lower variance?

Let $$\pi_\theta$$ be a target policy and $$\beta_\theta$$ be a behavior policy. I have seen the following 2 policy gradient estimators in the literature:

$$\operatorname*{E}_{\tau \sim \beta_\theta} \sum_t R_t \nabla_\theta \ln \pi_\theta(a_t | s_t)$$

where

$$R_t = \sum_{t' \geq t} r_{t'} \prod_{t'' \leq t'} \frac{\pi_\theta(a_{t''} | s_{t''})}{\beta_\theta(a_{t''} | s_{t''})}$$

and where

$$R_t = \sum_{t' \geq t} r_{t'} \prod_{t'' \leq t} \frac{\pi_\theta(a_{t''} | s_{t''})}{\beta_\theta(a_{t''} | s_{t''})}$$

The difference is that $$t''$$ goes up to $$t'$$ in the 1st and up to only $$t$$ in the 2nd. Which of these estimators has lower variance, and how can this be proven?

• It may be a good idea to provide a link to the references where you saw these formulas.
– nbro
Mar 26 '21 at 0:17