Vanilla policy gradient algorithm (using baseline to reduce variance) acc to here (page 16)
Initialize policy parameter θ, baseline b
for iteration=1, 2, . . . do
Collect a set of trajectories by executing the current policy
At each timestep in each trajectory, compute
the return $R_{t}= \sum_{t'=t}^{T-1}\gamma^{t'-t}r_{t'}$
the advantage estimate $\hat{A}_{t} = R_{t} - b(s_{t})$
Re-fit the baseline, by minimizing $\lVert b(s_{t}) - R_{t} \rVert^{2}$
summed over all trajectories and timesteps.
Update the policy, using a policy gradient estimate $\hat{g}$,
which is a sum of terms $\nabla_{\theta}log\pi(a_{t}|s_{t},\theta)\hat{A_{t}}$
- At line 6, advantage estimate is computed by subtracting baseline from the returns
- At line 7, baseline is re-fit minimizing mean squared error between state dependent baseline and return
- At line 8, we update the policy using advantage estimate from line 6
So is the baseline expected to be used in the next iteration when our policy has changed?
To compute the advantage we subtract the state value $V(s_{t})$ from the action value $Q(s_{t},a_{t})$, under the same policy, then why is the old baseline used here in advantage estimation?