1
$\begingroup$

Source: CS285 2023Fall homework2 analysis. enter image description here

In the part B, the expectation $\mathbb{E}[R(\tau)] = \theta + \theta^2 ... = \theta/(1-\theta)$. Derivate this with respect to $\theta$ is $1/(1-\theta)^2$. But how to compute the gradient using policy gradients?(Part A)

enter image description here

I'm really confused for computing the expectation of the gradient of log of $\pi_\theta$(Trajectory).

$\endgroup$
2
  • $\begingroup$ Hi, welcome to the site! What's the source of your first picture? And the source of the second one? $\endgroup$
    – evaristegd
    Commented Mar 24 at 15:06
  • $\begingroup$ Both pictures come from CS285 2023Fall homework2 analysis. I added this into question. : ) $\endgroup$
    – yeebo xie
    Commented Mar 25 at 3:08

1 Answer 1

3
$\begingroup$

For part B, from this simple state diagram you're right we can directly compute the expected return without discounting as $\mathbb{E}_{\tau \sim \pi_{\theta}}R(\tau)=\theta(1-\theta)+2\theta^2(1-\theta)+3\theta^3(1-\theta)+...=\theta+\theta^2+\theta^3+...=\theta/(1-\theta)$, where $|\theta|<1$. Then take its gradient w.r.t $\theta$ we simply arrive at $1/(1-\theta)^2$.

For part A it seems supposed to use policy gradient theorem for a generic MDP to calculate the same, which is usually expressed as $\nabla_{\theta}J(\theta)=\mathbb{E}_{\tau \sim \pi_{\theta}}[\nabla_{\theta}\log\pi_{\theta}(\tau)R(\tau)]$ following your reference's notation with a random trajectory $\tau$. Similar to part B above to calculate the expected return, we can simplify the policy gradient as $$\nabla_{\theta}J(\theta)=\theta(1-\theta)·\nabla_{\theta}\log\theta(1-\theta)·1+\theta^2(1-\theta)·\nabla_{\theta}\log\theta^2(1-\theta)·2+...=\nabla_{\theta}\theta(1-\theta)·1+\nabla_{\theta}\theta^2(1-\theta)·2+...=\nabla_{\theta}(\theta(1-\theta)·1+\theta^2(1-\theta)·2+...)=\nabla_{\theta}(\theta/(1-\theta))$$, which obviously matches the result of the above Part B. Note here my answer didn't use or need the hint of your reference since only elementary convergent geometric series sum formula and basic calculus formula involving logarithm and division are actually involved.

$\endgroup$
10
  • $\begingroup$ Thanks for the answer which really helps a lot! $\endgroup$
    – yeebo xie
    Commented Mar 24 at 10:13
  • $\begingroup$ By MDP, do you mean Markov Decision Process? And, what does the letter J stand for? $\endgroup$
    – evaristegd
    Commented Mar 24 at 15:13
  • 1
    $\begingroup$ @evaristegd J os the same as in your question, the loss $\endgroup$
    – Alberto
    Commented Mar 24 at 18:55
  • 1
    $\begingroup$ @evaristegd section 8 rail.eecs.berkeley.edu/deeprlcourse/deeprlcourse/static/… $\endgroup$
    – Alberto
    Commented Mar 24 at 21:51
  • 1
    $\begingroup$ @evaristegd exact match on google, just search "and verify that this matches the policy gradient" on google with quotes and you will find it $\endgroup$
    – Alberto
    Commented Mar 25 at 8:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .