1
$\begingroup$

I understand the general idea behind the Actor-Critic architecture. The actor maps state to action, and the critic maps state + action to reward.

But I don't fully understand how the critic output (reward space) can be used to train the actor sub-network.

How can a reward be used to train the actor sub-network to make it choose a better action when the actor sub-network doesn't get the reward as input or output?

$\endgroup$

1 Answer 1

1
$\begingroup$

Instead of receiving total return in the summation form of (discounted) rewards sampled from the full trajectory as in REINFORCE with baseline to update policy parameters, in actor-critic method to bootstrap, the actor sub-network usually receives a one-step return in the familiar form of temporal difference (TD) error $(R_{t+1} + \gamma \hat V_{\pi_\theta}(S_{t+1}, w) - \hat V_{\pi_\theta}(S_t,w))$ (or n-step) from the critic who evaluates estimated state value function optimized from same TD error.

Obviously from the computational point of view the policy gradient's "advantage" function from the critic's one-step or n-step TD error has much lower variance than that of REINFORCE with baseline from its sampled trajectory's total return, even if the former introduces bias.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .