4
$\begingroup$

What is the difference between vanilla policy gradient (VPG) with a baseline as value function and advantage actor-critic (A2C)?

By vanilla policy gradient I am specifically referring to spinning up's explanation of VPG.

$\endgroup$

1 Answer 1

5
$\begingroup$

The difference between Vanilla Policy Gradient (VPG) with a baseline as value function and Advantage Actor-Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA:

  • The value estimates used in updates for VPG are based on full sampled returns, calculated at the end of episodes.

  • The value estimates used in updates for A2C are based on temporal difference bootstrapped from e.g. a single step difference, and the Bellman function.

This leads to the following practical differences:

  • A2C can learn during an episode which can lead to faster refinements in policy than with VPG.

  • A2C can learn in continuing environments, whilst VPG cannot.

  • A2C relies on initially biased value estimates, so can take more tuning to find hyperparameters for the agent that allows for stable learning. Whilst VPG typically has higher variance and can require more samples to achieve the same degree of learning.

$\endgroup$
6
  • $\begingroup$ Thanks for your answer! What does "continuing environments" mean? $\endgroup$
    – Iyeeke
    Oct 5, 2020 at 17:27
  • 2
    $\begingroup$ @lyeeke: It's another way of saying "non-episodic" i.e. the environment has no terminal states and keeps going with no natural break points. $\endgroup$ Oct 5, 2020 at 17:37
  • $\begingroup$ It there anything inbetween? Like having this bootstrapped critic evaluation after every step but being able to adjust in the end of the episode if/when it turns out the bootstrapped value was wrong? $\endgroup$
    – hypers
    Dec 8, 2021 at 7:17
  • $\begingroup$ @hypers: You could use eligibility traces with the critic, it can be trained in almost any value-based way. Unfortunately you cannot easily go back and adjust the advice it gave to the actor in previous steps. $\endgroup$ Dec 8, 2021 at 8:38
  • $\begingroup$ I see from this TF Actor Critic article (tensorflow.org/tutorials/reinforcement_learning/actor_critic) that A2C can also be trained in the end of the episode (as done in that tutorial). So my reckoning is that A2C is not necessarily different from VPG by this criteria (when the training is happening). $\endgroup$
    – hypers
    Dec 8, 2021 at 10:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .