In the proof of the policy gradient theorem in the RL book of Sutton and Barto (that I shamelessly paste here):

enter image description here

there is the "unrolling" step that is supposed to be immediately clear

With just elementary calculus and re-arranging of terms

Well, it's not. :) Can someone explain this step in more detail?

How exactly is $Pr(s \rightarrow x, k, \pi)$ deduced by "unrolling"?


It looks like "v of s prime" is just substituted with the already derived value for "v of s". You can call it a recursion of a kind. In other words, v(s) is dependent on v(s') and that implies that v(s') is dependent on v(s''). So we can combine that and get the dependency of v(s) of v(s'').

| improve this answer | |
  • $\begingroup$ Where exactly does $Pr(s\rightarrow x, k, \pi)$ come from? $\endgroup$ – tmaric Jun 23 at 8:51
  • 1
    $\begingroup$ This is in the right direction for the first unrolling line, but actually the substituted term is $\nabla v_\pi(s')$ - you can use LaTex to put that into the answer - i.e. $\nabla v_\pi(s')$ $\endgroup$ – Neil Slater Jun 23 at 8:52
  • $\begingroup$ I see where the unrolling leads, in terms of simply recursively expanding the expression. What I don't see, is how $Pr$ is substituted in the unrolled expression. $\endgroup$ – tmaric Jun 23 at 9:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.