In vanilla policy gradient is the baseline lagging behind the policy?

Question

Vanilla policy gradient algorithm (using baseline to reduce variance) acc to here (page 16)

Initialize policy parameter θ, baseline b

for iteration=1, 2, . . . do

Collect a set of trajectories by executing the current policy

At each timestep in each trajectory, compute

the return $R_{t}= \sum_{t'=t}^{T-1}\gamma^{t'-t}r_{t'}$

the advantage estimate $\hat{A}_{t} = R_{t} - b(s_{t})$

Re-fit the baseline, by minimizing $\lVert b(s_{t}) - R_{t} \rVert^{2}$

summed over all trajectories and timesteps.

Update the policy, using a policy gradient estimate $\hat{g}$,

which is a sum of terms $\nabla_{\theta}log\pi(a_{t}|s_{t},\theta)\hat{A_{t}}$

• At line 6, advantage estimate is computed by subtracting baseline from the returns
• At line 7, baseline is re-fit minimizing mean squared error between state dependent baseline and return
• At line 8, we update the policy using advantage estimate from line 6

So is the baseline expected to be used in the next iteration when our policy has changed?

To compute the advantage we subtract the state value $V(s_{t})$ from the action value $Q(s_{t},a_{t})$, under the same policy, then why is the old baseline used here in advantage estimation?

Answer 1

So is the baseline expected to be used in the next iteration when our policy has changed?

Yes.

To compute the advantage we subtract the state value $V(s_{t})$ from the action value $Q(s_{t},a_{t})$, under the same policy, then why is the old baseline used here in advantage estimation?

The precise value of the baseline is not that important. What is important is that the baseline does not depend on the action choice, $a$, so it does not impact the gradient estimations or update steps for the policy function you are trying to improve.

You could in theory use a fixed offset instead of $V(s)$, or any arbitrary function that does not depend on $a$. In some settings the average reward $\bar{R}$ seen so far is used.

Using a rough approximation to $V(s)$ - and thus an approximate advantage function overall is useful as it removes a large source of variance in gradient estimates (the inherent value of the current state under the current policy, which is irrelevant to search for adjustments to that policy). The more accurate $V(s)$ is, the lower variance, thus faster and more reliable convergence, so you do want it to be a good estimate. But a little bit of lag behind policy updates is acceptable and does not break the algorithm.

For more on this, see Sutton & Barto, chapter 13, section 13.4.

Answer 2

To be honest I was also struggling with this question for quite some time and I think that it was written this way mostly because of historic reasons. Once you have the advantage $A_t = \sum_{i=t}^{T} r_{i+1} - V_\phi(s)$, you can use it directly to update both the value and the policy networks:

\begin{align} &w = w + \alpha A_t \nabla V_\phi(s) \\ &\theta = \theta + \alpha A_t \nabla \log \pi_\theta(a|s) \end{align}

To see why this is true note that you update the value network using the squared-error loss $L^{VF} = 0.5(\sum r_{t+1} - V_\phi(s))^2$, and when you differentiate this you get simply $A_t$. And the loss for the policy network is simply $L^{PG} = \log \pi_\theta(a|s) A_t$.

Essentially you evaluate the value network only one time, instead of two times, which probably back at the time when this algorithm was invented (e.g. in the 1990s) was a big deal.

If I were to implement vpg with baseline, I would first update the value network and only then calculate the advantage to update the policy network, i.e. I would reshuffle the steps so that I use the updated baseline. However, I would never use vpg with baseline. I don't think anyone has ever used it for something practical and I don't know of any papers that show SotA results with it.

Note that if you have the value network, instead of using it merely as a baseline, you would be much better off using it to bootstrap the estimate of the return, i.e. $A_t = r_{t+1} + V(s_{t+1}) - V(s_t).$
If you take a look at the pseudocode for this algorithm you can see that, again, people update the value network after they compute the advantage. Here, you could again update the value network before that, but now you have to be extra careful. When computing the advantage you have to use the old value network for bootstrapping and the new value network only for the baseline: $A_t = r_{t+1} + V_{old}(s_{t+1}) - V_{new}(s_t).$ A very brief explanation for this is given at the end of page 55 in John Schulman's phd thesis that you shared. So in this case, I would just go on and use the old network and not overthink it.

Even better than a one-step bootstrap, you could do an n-step bootstrap: $A_t^{(n)} = r_{t+1} + r_{t+2} + \cdots + r_{t+n} + V(s_{t+n}) - V(s_t)$ and then use a generalized advantage estimation $A^{GAE} = \sum_{n=1}^{\inf} (1-\lambda) \lambda ^ {n-1} A_{t}^{(n)}$. When computing $A^{GAE}$ the value network is used both for bootstrapping and as a baseline and it is not possible to distinguish between the two uses because they are cancelled out in a telescoping sum. This means that the only correct way would be to update the value network after you compute the advantage.

So you can see that once you have the value network you get from a poor algorithm to state-of-the-art with one simple equation. See here if you want to read more.