Let $\pi_\theta$ be a target policy and $\beta_\theta$ be a behavior policy. I have seen the following 2 policy gradient estimators in the literature:

$$ \operatorname*{E}_{\tau \sim \beta_\theta} \sum_t R_t \nabla_\theta \ln \pi_\theta(a_t | s_t) $$


$$ R_t = \sum_{t' \geq t} r_{t'} \prod_{t'' \leq t'} \frac{\pi_\theta(a_{t''} | s_{t''})}{\beta_\theta(a_{t''} | s_{t''})} $$

and where

$$ R_t = \sum_{t' \geq t} r_{t'} \prod_{t'' \leq t} \frac{\pi_\theta(a_{t''} | s_{t''})}{\beta_\theta(a_{t''} | s_{t''})} $$

The difference is that $t''$ goes up to $t'$ in the 1st and up to only $t$ in the 2nd. Which of these estimators has lower variance, and how can this be proven?

  • $\begingroup$ It may be a good idea to provide a link to the references where you saw these formulas. $\endgroup$
    – nbro
    Mar 26 at 0:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.