# Why is the "reward to go" replaced by Q instead of V, when transitioning from PG to actor critic methods?

While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see this slide 5).

I am not able to understand how this is mathematically justified. It seems intuitive to me that the "reward to go" when sampled through multiple trajectories should be estimated by the state-value function.

I feel this way since nowhere in the objective function formulation or resulting gradient expression do we tie down the first action after reaching a state. Alternatively, when we sample a bunch of trajectories, these trajectories might include different actions being taken from the state reached in timestep $$t$$.

So, why isn't the estimation/approximation for the "reward to go" the state value function, in which the expectation is also over all the actions that may be taken from that state as well?

When you say simple policy gradient, I assume you mean something like REINFORCE.

The main difference between actor-critic and REINFORCE-like algorithms is in how they estimate the reward to go:

• in REINFORCE, you wait until a trajectory terminates to make any updates, and your estimator of the reward to go is the actual reward to go that was observed in the trajectory.

• in actor critic algorithms, instead of using an entire trajectory to estimate a reward to go, you use the action value function (note that this way, you can compute updates at every time step). The reason why you use the action-value function is because at each transition, you have already committed to some action.

The benefit of the actor critic estimator is that it exhibits much less variance. REINFORCE estimators are Monte Carlo estimators, which are known to exhibit extremely high variance.

Now, the actor critic method in the slides you posted takes variance reduction one step further. Instead of estimating reward to go, it's estimating advantage -- that is, the difference between the current action value and your estimated state value. The state value function term doesn't depend on the action, so it doesn't affect the expected value of the policy gradient. However, it serves as a 'baseline', which helps reduce the variance of the reward to go estimator even further.

• I see. Just to be clear, in REINFORCE is there a single update after a trajectory terminates or are there T updates, where for each timestep the gradient is approximated as grad log prob of policy at that timestep multiplied by reward to go from that timestep? Also then in the case of on policy actor critic, the advantage is Q-V is given as r + gammaV'. But then we also try to get V to converge to r+gammaV'. Could you please elaborate a bit more on the mathematical justification in using Q and V? Jun 15, 2020 at 16:06
• Firstly, in REINFORCE, there should be T updates after each trajectory as you suggested Jun 15, 2020 at 16:26
• For your second question, especially in the on-policy setting, you need to account for the action you took (V doesn't do this). Let's say you just observed the following transition, (s, a, r, s'). You want to use this transition to approximate the reward to go. To do this with the data you have, you're assuming the next state is s', but this is correlated with the fact that you took action a. That's why you need to use Q instead of, V, otherwise r + gamma * V(s') wouldn't necessarily be a valid estimate of the reward to go. Jun 15, 2020 at 16:29
• Okay. But I am still unable to understand why the advantage goes to zero once the critic gets better. We seem to set the target for V as r + gamma*V'. But the difference between the two is what we call Advantage Jun 16, 2020 at 3:27
• Right, because once the critic and policy have converged, the best actions are chosen by the policy, so the Q value should equal V (so advantage is 0). Jun 16, 2020 at 3:46