What is the difference between vanilla policy gradient (VPG) with a baseline as value function and advantage actor-critic (A2C)?

By vanilla policy gradient I am specifically referring to spinning up's explanation of VPG.


The difference between Vanilla Policy Gradient (VPG) with a baseline as Value function and Advantage Actor Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA:

  • The value estimates used in updates for VPG are based on full sampled returns, calculated at the end of episodes.

  • The value estimates used in updates for A2C are based on temporal difference bootstrapped from e.g. a single step difference, and the Bellman function.

This leads to following practical differences:

  • A2C can learn during an episode which can lead to faster refinements in policy than with VPG.

  • A2C can learn in continuing environments, whilst VPG cannot.

  • A2C relies on initially biased value estimates, so can take more tuning to find hyperparameters for the agent that allow for stable learning. Whilst VPG typically has higher variance and can require more samples to achieve the same degree of learning.

  • $\begingroup$ Thanks for your answer! What does "continuing environments" mean? $\endgroup$
    – Iyeeke
    Oct 5 '20 at 17:27
  • 1
    $\begingroup$ @lyeeke: It's another way of saying "non-episodic" i.e. the environment has no terminal states and keeps going with no natural break points. $\endgroup$ Oct 5 '20 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.