1
$\begingroup$

I understand the general idea behind the Actor-Critic architecture. The actor maps state to action, and the critic maps state + action to reward.

But I don't fully understand how the critic output (reward space) can be used to train the actor sub-network.

How can a reward be used to train the actor sub-network to make it choose a better action when the actor sub-network doesn't get the reward as input or output?

$\endgroup$
1
  • $\begingroup$ As shown in my below answer, the only difference between actor-critic method and REINFORCE with baseline is the former also applies the same state value estimator to the next state of any state transition to form the TD error and thus is a kind way of assessing the latest done action before the state transition to further update actor network parameters, while the latter's state value estimator doesn't assess the action taken at any same iteration during an episode via any of the next state value at all to influence its next action. $\endgroup$
    – mohottnad
    13 hours ago

1 Answer 1

1
$\begingroup$

Instead of receiving total return in the summation form of (discounted) rewards sampled from the full trajectory as in REINFORCE with baseline to update policy parameters, in actor-critic method to bootstrap, the actor sub-network usually receives a one-step return in the familiar form of temporal difference (TD) error $(R_{t+1} + \gamma \hat V_{\pi_\theta}(S_{t+1}, w) - \hat V_{\pi_\theta}(S_t,w))$ (or n-step) from the critic who evaluates estimated state value function optimized from same TD error.

Obviously from the computational point of view the policy gradient's "advantage" function from the critic's one-step or n-step TD error has much lower variance than that of REINFORCE with baseline from its sampled trajectory's total return, even if the former introduces bias.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .