What would be the reason for having a different network architecture for the actor vs. value function networks in PPO?

Question

I was reading this link , and saw some creative architectures for PPO.

I know the "No Free Lunch Theorem" and all, but what would be the logic/reasoning for why you would choose to have a different size/shape/depth for the actor vs. value func. networks in PPO?

I just came across this question, which is related but not the same. It asks about the utility of sharing parameters across the two networks, to which the answer is essentially "that might accelerate training by having less parameters". I think my question is distinct and more general.

Answer 1

I would usually go with two separate networks for several reasons:

1. Having a single network with two heads (one for policy and one for value) might be more challenging to train. I believe that training two separate networks is more stable. Especially in the case of PPO, you want to have small policy updates and you clip if the update would be too large. But now updating the value network also updates your policy, so you also have to clip the value loss as well, but what should be the clip size for the value net ? (see here for more)

2. In the case of a single network you have a single objective where you add the policy loss and the value loss: $L = L^{CLIP} + cL^{VF}$, and now you need to adjust the hyperparameter $c$. See Equation. 9 from the official PPO paper. Of course, if you have two networks you also have other hyper parameters, like different learning rates and so on, but these are not so much interdependent and can in general be adjusted separately, i.e. if your value net is training well but the policy is not, then don't touch the hyperparams of the value net and only adjust the hyperparams for the policy net.

3. The architecture of the two networks doesn't have to be the same. I usually use a much simpler network for the value network and a more complex model for the policy network. The task that the policy network is learning is much harder and at the end when all training is done you are actually using only this network. On the other hand your value network is trained on a regression task, i.e. predicting a single number, which is much easier and also better understood theoretically.

4. You might also want to regularize the models differently. For example I would regularize the value network using L2-regularization, but I would not use this for the policy network. For the policy network I would use entropy regularization.