How long should the state-dependent baseline for policy gradient methods be trained at each iteration?

Question

How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods?

I'm using this equation to compute the policy gradient:

$$ \nabla_{\theta} J\left(\pi_{\theta}\right)=\underset{\tau \sim \pi_{\theta}}{\mathrm{E}}\left[\sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}\left(a_{t} | s_{t}\right)\left(\sum_{t^{\prime}=t}^{T} R\left(s_{t^{\prime}}, a_{t^{\prime}}, s_{t^{\prime}+1}\right)-b\left(s_{t}\right)\right)\right] $$

Here is mentioned to use one or more gradient steps, so is it a hyper-parameter to be found using random search?

Is there some way we can use an adaptive method to find out when to stop?

In an experiment to train Cartpole-v2 using a policy gradient with baseline, I found the results are better when applying 5 updates than when only a single update was applied.

Note: I am referring to the number of updates to take on a single batch of q values encountered across trajectories collected using current policy.

Answer 1

I went into the pytorch code for the spinning up implementation of vanilla policy gradient and from what I could understand, found that they use a learning rate of 1e-3 for training the baseline and do a gradient descent 80 times on the same dataset by default with no termination criteria.

Also it is usually impossible to fit the value function completely as we are using a function approximator. The main point is not to get too carried away to get the lowest loss as the main improvement in the agent's performance will happen by doing policy gradient descents rather than trying to aggressively get the accurate value function for the baseline.

Link for the implementation: spinning up implementation of vpg