# What is the rigorous proof behind the causality simplification step of off-policy policy gradient

In lec5 of Sergey Levine's CS258 on RL, I can understand the derivation of the causality simplification of on-polciy version below (page 17).

But what is the detial of the derivation of the last line in the PPT below (page 25)?

It seems that the cumulative reward term and the importance ratio term cannot be decompled and handled one by one.

• casuality =/= causality Commented Dec 31, 2023 at 20:21
• @Alberto Thanks for your kind reminder. Have edited. Commented Jan 1 at 2:24

1. Bring the distribution from $$\theta$$ to $$\theta '$$ to cancel the importance ratio term. Use causality to simplify the cumulative reward term to reward-to-go. Details can be found here, which utilized Expected Grad-Log-Prob Lemma.
2. Bring the distribution back to $$\theta$$. Use distribution law and the definition of marginal distribution to get the final formula.
\begin{align} &\mathop{\mathbb{E}}_{\tau\sim \theta}\left[ \left( \prod_{t=1}^{T} \frac{\pi_{\theta '}(a_t|s_t)}{\pi_{\theta }(a_t|s_t)} \right)\left(\sum_{t=1}^{T}\nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t) \right) \sum_{t=1}^{T}r(s_t,a_t)\right] \\ =&\mathop{\mathbb{E}}_{\tau\sim \theta '}\left[\left(\sum_{t=1}^{T}\nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)\right) \sum_{t=1}^{T}r(s_t,a_t)\right] \\ =& \mathop{\mathbb{E}}_{\tau\sim \theta '}\left[\sum_{t=1}^{T}\left(\nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t) \sum_{t'=t}^{T}r(s_{t'},a_{t'})\right)\right] \tag{causility} \\ =& \sum_{t=1}^{T} \sum_{t'=t}^{T} \mathop{\mathbb{E}}_{\tau\sim \theta '} \left[\nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)r(s_{t'},a_{t'}) \right] \tag{distribution law}\\ =& \sum_{t=1}^{T} \sum_{t'=t}^{T} \mathop{\mathbb{E}}_{\tau\sim \theta} \left[\prod_{{t''}=1}^{T} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})} \nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)r(s_{t'},a_{t'}) \right]\\ =& \sum_{t=1}^{T} \sum_{t'=t}^{T} \mathop{\mathbb{E}}_{\tau\sim \theta} \left[\prod_{{t''}=1}^{t'} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})} \nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)r(s_{t'},a_{t'}) \right] \tag{marginal distribution}\\ =& \mathop{\mathbb{E}}_{\tau\sim \theta} \left[\sum_{t=1}^{T} \nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)\sum_{t'=t}^{T} \prod_{{t''}=1}^{t'} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})} r(s_{t'},a_{t'}) \right]\\ =& \mathop{\mathbb{E}}_{\tau\sim \theta} \left[\sum_{t=1}^{T} \nabla_{\theta'}\log \pi_{\theta '}(a_t|s_t)\prod_{{t''}=1}^{t} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})}\sum_{t'=t}^{T} r(s_{t'},a_{t'})\prod_{{t''}=t}^{t'} \frac{\pi_{\theta '}(a_{t''}|s_{t''})}{\pi_{\theta }(a_{t''}|s_{t''})} \right] \\ \end{align}