2
$\begingroup$

The PPO objective may include a value function error term when parameters are shared between the policy and value functions. How does this help, and when to use a neural network architecture that shares parameters between the policy and value functions, as opposed to two neural networks with separate parameters?

$\endgroup$
0
$\begingroup$

Think of the network as a feature extractor followed by a policy head and a value function head. The feature extractor compresses the inputs into a lower dimensional feature vector that we hypothesize will be useful to both the policy and the value function. Then you train the policy and value function with those learned features as inputs (which is in theory much easier than training both with higher dimensional inputs). This way you save a bunch of parameters and hopefully accelerate training.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.