In vanilla policy gradient is the baseline lagging behind the policy?

Question

Vanilla policy gradient algorithm (using baseline to reduce variance) acc to here (page 16)

Initialize policy parameter θ, baseline b

for iteration=1, 2, . . . do

Collect a set of trajectories by executing the current policy

At each timestep in each trajectory, compute

the return $R_{t}= \sum_{t'=t}^{T-1}\gamma^{t'-t}r_{t'}$

the advantage estimate $\hat{A}_{t} = R_{t} - b(s_{t})$

Re-fit the baseline, by minimizing $\lVert b(s_{t}) - R_{t} \rVert^{2}$

summed over all trajectories and timesteps.

Update the policy, using a policy gradient estimate $\hat{g}$,

which is a sum of terms $\nabla_{\theta}log\pi(a_{t}|s_{t},\theta)\hat{A_{t}}$

• At line 6, advantage estimate is computed by subtracting baseline from the returns
• At line 7, baseline is re-fit minimizing mean squared error between state dependent baseline and return
• At line 8, we update the policy using advantage estimate from line 6

So is the baseline expected to be used in the next iteration when our policy has changed?

To compute the advantage we subtract the state value $V(s_{t})$ from the action value $Q(s_{t},a_{t})$, under the same policy, then why is the old baseline used here in advantage estimation?

Answer 1

So is the baseline expected to be used in the next iteration when our policy has changed?

Yes.

To compute the advantage we subtract the state value $V(s_{t})$ from the action value $Q(s_{t},a_{t})$, under the same policy, then why is the old baseline used here in advantage estimation?

The precise value of the baseline is not that important. What is important is that the baseline does not depend on the action choice, $a$, so it does not impact the gradient estimations or update steps for the policy function you are trying to improve.

You could in theory use a fixed offset instead of $V(s)$, or any arbitrary function that does not depend on $a$. In some settings the average reward $\bar{R}$ seen so far is used.

Using a rough approximation to $V(s)$ - and thus an approximate advantage function overall is useful as it removes a large source of variance in gradient estimates (the inherent value of the current state under the current policy, which is irrelevant to search for adjustments to that policy). The more accurate $V(s)$ is, the lower variance, thus faster and more reliable convergence, so you do want it to be a good estimate. But a little bit of lag behind policy updates is acceptable and does not break the algorithm.

For more on this, see Sutton & Barto, chapter 13, section 13.4.