Source: CS285 2023Fall homework2 analysis.

In the part B, the expectation $$\mathbb{E}[R(\tau)] = \theta + \theta^2 ... = \theta/(1-\theta)$$. Derivate this with respect to $$\theta$$ is $$1/(1-\theta)^2$$. But how to compute the gradient using policy gradients?(Part A)

I'm really confused for computing the expectation of the gradient of log of $$\pi_\theta$$(Trajectory).

• Hi, welcome to the site! What's the source of your first picture? And the source of the second one? Commented Mar 24 at 15:06
• Both pictures come from CS285 2023Fall homework2 analysis. I added this into question. : ) Commented Mar 25 at 3:08

For part B, from this simple state diagram you're right we can directly compute the expected return without discounting as $$\mathbb{E}_{\tau \sim \pi_{\theta}}R(\tau)=\theta(1-\theta)+2\theta^2(1-\theta)+3\theta^3(1-\theta)+...=\theta+\theta^2+\theta^3+...=\theta/(1-\theta)$$, where $$|\theta|<1$$. Then take its gradient w.r.t $$\theta$$ we simply arrive at $$1/(1-\theta)^2$$.
For part A it seems supposed to use policy gradient theorem for a generic MDP to calculate the same, which is usually expressed as $$\nabla_{\theta}J(\theta)=\mathbb{E}_{\tau \sim \pi_{\theta}}[\nabla_{\theta}\log\pi_{\theta}(\tau)R(\tau)]$$ following your reference's notation with a random trajectory $$\tau$$. Similar to part B above to calculate the expected return, we can simplify the policy gradient as $$\nabla_{\theta}J(\theta)=\theta(1-\theta)·\nabla_{\theta}\log\theta(1-\theta)·1+\theta^2(1-\theta)·\nabla_{\theta}\log\theta^2(1-\theta)·2+...=\nabla_{\theta}\theta(1-\theta)·1+\nabla_{\theta}\theta^2(1-\theta)·2+...=\nabla_{\theta}(\theta(1-\theta)·1+\theta^2(1-\theta)·2+...)=\nabla_{\theta}(\theta/(1-\theta))$$, which obviously matches the result of the above Part B. Note here my answer didn't use or need the hint of your reference since only elementary convergent geometric series sum formula and basic calculus formula involving logarithm and division are actually involved.