I am currently learning policy gradient methods from the Deep RL boot camp by Pieter Abbeel in which he explains the actor-critic algorithm derivation.

At around minute 39, he explains that the sum of the rewards from time step $t$ onwards is actually an estimation of $Q^\pi(s,u)$. I understand the definition of $Q^\pi(s,u)$ but I'm not sure why this is the case here. Is the reward following after time step $t+1$ collected based on current policy?

enter image description here

  • $\begingroup$ yes, you are getting rewards following the policy $\pi$. After completing each trajectory, then you will update the policy $\pi$. $\endgroup$ – Swakshar Deb May 10 '20 at 17:40
  • $\begingroup$ so at state $s_k^(i)$ the action taken is $u_k$ but for state $s_{k+1}$ onwards, the policy $\pi$ is being followed ? but the notation does not contain anything about $\pi$ in the $R(s_k,u_k)$ term $\endgroup$ – calveeen May 11 '20 at 2:22
  • $\begingroup$ The notation $R(s_{k},u_{k})$ is not clear, but the rewards you get is following your current policy $\endgroup$ – Swakshar Deb May 11 '20 at 5:39
  • $\begingroup$ but why would that be an estimation of Q ? if following the current policy it should be $V^\pi$ ? $\endgroup$ – calveeen May 11 '20 at 7:03
  • $\begingroup$ You are sampling action($u_{k}$) from the current policy, $u_{k} \sim \pi$. So, it is like that, you are at the state $s_{k}$, from that state sample action $u_{k}$ from current policy $\pi$, thereafter following the policy $\pi$. By defination this is action value function ($Q^{\pi}(s_{k},u_{k})$). Don't be confused with: $Q^{\pi}(s_{t},\pi(a_{t}|s_{t})$ and $Q^{\pi}(s,a)$ where $a \sim \pi$, those two have different meaning. $\endgroup$ – Swakshar Deb May 11 '20 at 7:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.