# Why Phasic Policy Gradient (PPG) can update value function in auxiliary phase?

My questions is that how could we train the value network (separated from shared network) by using data from previous policies, which varies a lot since we collect data from different policies with many training phases done to start a auxiliary phase, especially returns are calculated by those policies. Wouldn't it hurt the stability of fitting value function?

## 1 Answer

PPG can update the value function in the auxiliary phase because it uses a technique called "reparameterization" which allows for the gradient of the value function to be computed more efficiently. Additionally, PPG can take advantage of the structure of the problem to provide a better estimate of the value function.

I think you can train the value network on data from any policy, but it will have more variance. If you have access to more data, it could help to smooth out the value function.

One approach would be to update the value network after each training phase of the policy network. This would ensure that the value network is always fitted to the current policy. However, this might not be practical if the training phases are very long.

Another approach would be to fit the value network using a rolling window of data from the previous training phases. This would strike a balance between stability and accuracy.

"Wouldn't it hurt the stability of fitting value function?"$$\rightarrow$$ $$\color{red}{\textbf{No}}$$, it should not hurt the stability of fitting the value function. In fact, using data from different policies can actually help to improve the stability of the value function estimate. This is because of "reparameterization" which allows for the gradient of the value function to be computed more efficiently as discussed above.

• Aren't we estimating the value function w.r.t $\pi$? If we fit the value function with returns coming from different policies, we are moving towards different $V^{\pi}$? Commented Oct 18, 2022 at 3:59