Following Pytorch's actor critic, I understand that the critic is a function mapping from the state space to the reward space, meaning, the critic approximates the state-value funcion.

However, according to This paper (you don't need to read it, just a glance at the nice picture at page 2 is enough), the critic is a function mapping from the action space to the reward, meaning it approximates the action value funcion

I am confused.

When people say "actor critic" - what do they mean by "critic"?

Is the term "Critic" ambiguous in RL?


The same argument of a state-value function int Reinforcement Learning: An Introduction Section 13.5 can be applied to state-action values. The main takeaway is that a critic's state value (or action-state value) function is used for bootstrapping.

Although the REINFORCE-with-baseline method learns both a policy and a state-value function, we do not consider it to be an actor–critic method because its state-value function is used only as a baseline, not as a critic. That is, it is not used for bootstrapping (updating the value estimate for a state from the estimated values of subsequent states), but only as a baseline for the state whose estimate is being updated. This is a useful distinction, for only through bootstrapping do we introduce bias and an asymptotic dependence on the quality of the function approximation. As we have seen, the bias introduced through bootstrapping and reliance on the state representation is often beneficial because it reduces variance and accelerates learning.

| improve this answer | |
  • $\begingroup$ Interesting, so do you mean that the pytorch implementation of actor-critic (github.com/pytorch/examples/blob/master/reinforcement_learning/…) is inaptly named and should rather sound something in the line of "REINFORCE with function approximator as baseline"? As I understand that is the case because we adjust the advantage based on the true returns (not bootstrapped) $\endgroup$ – d56 Jan 22 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.