Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I understand the general idea behind the Actor-Critic architecture. The actor maps state to action, and the critic maps state + action to reward.
But I don't fully understand how the critic output (reward space) can be used to train the actor sub-network.
How can a reward be used to train the actor sub-network to make it choose a better action when the actor sub-network doesn't get the reward as input or output?
Instead of receiving total return in the summation form of (discounted) rewards sampled from the full trajectory as in REINFORCE with baseline to update policy parameters, in actor-critic method to bootstrap, the actor sub-network usually receives a one-step return in the familiar form of temporal difference (TD) error $(R_{t+1} + \gamma \hat V_{\pi_\theta}(S_{t+1}, w) - \hat V_{\pi_\theta}(S_t,w))$ (or n-step) from the critic who evaluates estimated state value function optimized from same TD error.
Obviously from the computational point of view the policy gradient's "advantage" function from the critic's one-step or n-step TD error has much lower variance than that of REINFORCE with baseline from its sampled trajectory's total return, even if the former introduces bias.
