8
$\begingroup$

I'm struggling to understand the difference between Actor-Critic and Advantage Actor-Critic.

At least I know they are different from Asynchronous Advantage Actor-Critic (A3C), as A3C adds asynchronous mechanism that uses multiple worker agents interacting with their own copy of environment and report the gradient to the global agent.

But what is the difference from the Actor-Critic and Advantage Actor-Critic (A2C)? Is it simply with or without Advantage function? But then, does the Actor-Critic have any other implementation except the use of Advantage function?

Or maybe are they synonyms and Actor-Critic is just a shorthand for A2C?

$\endgroup$
  • $\begingroup$ Just informing you that, in hindsight, I think my old answer was not completely accurate, and I edited it to fix it now. $\endgroup$ – Dennis Soemers Feb 18 at 8:30
7
$\begingroup$

Actor-Critic is not just a single algorithm, it should be viewed as a "family" of related techniques. They're all techniques based on the policy gradient theorem, which train some form of critic that computes some form of value estimate to plug into the update rule as a lower-variance replacement for the returns at the end of an episode. They all perform "bootstrapping" by using some sort of prediction of value.

Advantage Actor-Critic specifically uses estimates of the advantage function $A(s, a) = V(s) - Q(s, a)$ for its bootstrapping, whereas "actor-critic" without the "advantage" qualifier is not specific; it could be a trained $V(s)$ function, it could be some sort of estimate of $Q(s, a)$, it could be a variety of things.

In practice, the critic of Advantage Actor-Critic methods actually can just be trained to predict $V(s)$. Combined with an empirically observed reward $r$, they can then compute the advantage estimate $A(s, a) = r + \gamma V(s') - V(s)$.

$\endgroup$
  • $\begingroup$ The linked article first mentions that in advantage actor critic, the multiplicative term of the policy update is Q instead of R (used in REINFORCE). With this definition, I thought that the critic in advantage actor critic would have to predict Q(s, a) but you say it predicts A(s, a), and then the article introduces A(s, a) = r + gamma V(s') - V(s), without explaining how this changes the update which used Q. Also, if A(s, a) can be approximated using only V, then we only need to learn V, and that'd be exactly your definition of actor-critic, so I'm not sure what the difference is. $\endgroup$ – Mei Zhang Feb 18 at 4:55
  • $\begingroup$ @MeiZhang You're right, looking back at my old answer I think it wasn't good. I edited it to be better now. $\endgroup$ – Dennis Soemers Feb 18 at 8:29
  • $\begingroup$ Nice clarification. Do you know an example of an actor-critic method that is not advantage actor-critic? Also, would it be fair to say that PPO is a type of advantage actor-critic method? $\endgroup$ – Mei Zhang Feb 18 at 10:06
  • $\begingroup$ @MeiZhang Some options are listed here, but I don't think they have clear names really. Advantages are certainly the most commonly-used. Sure, PPO could be viewed as an advantage actor-critic method. $\endgroup$ – Dennis Soemers Feb 18 at 10:35
1
$\begingroup$

According to Sutton and Barto, they are the same thing. Note 13.5-6 (page 338) of their Reinforcement Learning: An Introduction, 2nd Edition book:

"Actor–critic methods are sometimes referred to as advantage actor–critic methods in the literature."

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.