# Is the reward following after time step $t+1$ collected based on current policy?

I am currently learning policy gradient methods from the Deep RL boot camp by Pieter Abbeel in which he explains the actor-critic algorithm derivation.

At around minute 39, he explains that the sum of the rewards from time step $$t$$ onwards is actually an estimation of $$Q^\pi(s,u)$$. I understand the definition of $$Q^\pi(s,u)$$ but I'm not sure why this is the case here. Is the reward following after time step $$t+1$$ collected based on current policy?

• yes, you are getting rewards following the policy $\pi$. After completing each trajectory, then you will update the policy $\pi$. – Swakshar Deb May 10 '20 at 17:40
• so at state $s_k^(i)$ the action taken is $u_k$ but for state $s_{k+1}$ onwards, the policy $\pi$ is being followed ? but the notation does not contain anything about $\pi$ in the $R(s_k,u_k)$ term – calveeen May 11 '20 at 2:22
• The notation $R(s_{k},u_{k})$ is not clear, but the rewards you get is following your current policy – Swakshar Deb May 11 '20 at 5:39
• but why would that be an estimation of Q ? if following the current policy it should be $V^\pi$ ? – calveeen May 11 '20 at 7:03
• You are sampling action($u_{k}$) from the current policy, $u_{k} \sim \pi$. So, it is like that, you are at the state $s_{k}$, from that state sample action $u_{k}$ from current policy $\pi$, thereafter following the policy $\pi$. By defination this is action value function ($Q^{\pi}(s_{k},u_{k})$). Don't be confused with: $Q^{\pi}(s_{t},\pi(a_{t}|s_{t})$ and $Q^{\pi}(s,a)$ where $a \sim \pi$, those two have different meaning. – Swakshar Deb May 11 '20 at 7:53