# What is the difference between actor-critic and advantage actor-critic?

I'm struggling to understand the difference between actor-critic and advantage actor-critic.

At least, I know they are different from asynchronous advantage actor-critic (A3C), as A3C adds an asynchronous mechanism that uses multiple worker agents interacting with their own copy of the environment and reports the gradient to the global agent.

But what is the difference between the actor-critic and advantage actor-critic (A2C)? Is it simply with or without advantage function? But, then, does the actor-critic have any other implementation except for the use of advantage function?

Or maybe are they synonyms and actor-critic is just a shorthand for A2C?

Actor-Critic is not just a single algorithm, it should be viewed as a "family" of related techniques. They're all techniques based on the policy gradient theorem, which train some form of critic that computes some form of value estimate to plug into the update rule as a lower-variance replacement for the returns at the end of an episode. They all perform "bootstrapping" by using some sort of prediction of value.

Advantage Actor-Critic specifically uses estimates of the advantage function $$A(s, a) = V(s) - Q(s, a)$$ for its bootstrapping, whereas "actor-critic" without the "advantage" qualifier is not specific; it could be a trained $$V(s)$$ function, it could be some sort of estimate of $$Q(s, a)$$, it could be a variety of things.

In practice, the critic of Advantage Actor-Critic methods actually can just be trained to predict $$V(s)$$. Combined with an empirically observed reward $$r$$, they can then compute the advantage estimate $$A(s, a) = r + \gamma V(s') - V(s)$$.

• Nice clarification. Do you know an example of an actor-critic method that is not advantage actor-critic? Also, would it be fair to say that PPO is a type of advantage actor-critic method? – Mei Zhang Feb 18 '19 at 10:06
• @MeiZhang Some options are listed here, but I don't think they have clear names really. Advantages are certainly the most commonly-used. Sure, PPO could be viewed as an advantage actor-critic method. – Dennis Soemers Feb 18 '19 at 10:35
• Can advantage actor critic use the loss gradient $$-\sum_t \left( Q_\phi(s_t, a_t) - \sum_a \pi_\theta(s_t, a) Q_\phi(s_t, a) \right) \nabla_\theta \log \pi_\theta(s_t, a_t) + \nabla_\phi \frac{1}{T} \sum_t (R_t - Q_\phi(s_t, a_t))^2$$ (where $R_t$ are the discounted returns), as described in the Mean Actor Critic paper? It's not doing well in my implementation, for some reason. I can post a separate question if you'd like. – user76284 Oct 12 '20 at 22:17
• @user76284 I didn't carefully inspect the equation you wrote there, but I'm familiar with the paper and yes, I see no reason why that wouldn't work. I've used something with similar intuition (summing up over all the actions, multiplying predicted values with policy's probabilities) in arxiv.org/abs/1905.05809. That was a very different setting though, games instead of single-agent RL, with value estimates produced by MCTS instead of a network. That seemed to work fine though. – Dennis Soemers Oct 13 '20 at 9:30