What is the difference between vanilla policy gradient (VPG) with a baseline as value function and advantage actor-critic (A2C)?
By vanilla policy gradient I am specifically referring to spinning up's explanation of VPG.
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
6
The difference between Vanilla Policy Gradient (VPG) with a baseline as value function and Advantage Actor-Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA:
The value estimates used in updates for VPG are based on full sampled returns, calculated at the end of episodes.
The value estimates used in updates for A2C are based on temporal difference bootstrapped from e.g. a single step difference, and the Bellman function.
This leads to the following practical differences:
A2C can learn during an episode which can lead to faster refinements in policy than with VPG.
A2C can learn in continuing environments, whilst VPG cannot.
A2C relies on initially biased value estimates, so can take more tuning to find hyperparameters for the agent that allows for stable learning. Whilst VPG typically has higher variance and can require more samples to achieve the same degree of learning.
answered Jul 27, 2020 at 10:07
-
$\begingroup$ Thanks for your answer! What does "continuing environments" mean? $\endgroup$– IyeekeOct 5, 2020 at 17:27
-
3$\begingroup$ @lyeeke: It's another way of saying "non-episodic" i.e. the environment has no terminal states and keeps going with no natural break points. $\endgroup$ Oct 5, 2020 at 17:37
-
$\begingroup$ It there anything inbetween? Like having this bootstrapped critic evaluation after every step but being able to adjust in the end of the episode if/when it turns out the bootstrapped value was wrong? $\endgroup$– hypersDec 8, 2021 at 7:17
-
$\begingroup$ @hypers: You could use eligibility traces with the critic, it can be trained in almost any value-based way. Unfortunately you cannot easily go back and adjust the advice it gave to the actor in previous steps. $\endgroup$ Dec 8, 2021 at 8:38
-
$\begingroup$ I see from this TF Actor Critic article (tensorflow.org/tutorials/reinforcement_learning/actor_critic) that A2C can also be trained in the end of the episode (as done in that tutorial). So my reckoning is that A2C is not necessarily different from VPG by this criteria (when the training is happening). $\endgroup$– hypersDec 8, 2021 at 10:57
$\begingroup$
$\endgroup$
Given the formula for the policy gradient with baseline:
$$ \nabla J(\theta) = \mathbb{E}_{a,s \sim \pi_\theta} \bigg[ \nabla \log \pi_\theta(a|s) \Big(R(s, a) - V_\phi(s) \Big) \bigg] $$
How do you compute the return $R(s,a)$ ?
If you use a simple Monte-Carlo estimate, i.e. $R = \sum_{t=t'}^{T} r_{t+1}$, then you get the "vpg with baseline" as it is called in the spinning up documentation. Note that in this case you have to rollout episode until it is finished, otherwise you can't compute the return !
If you use a one-step bootstrapped estimate, i.e. $ R(s,a) = r + V_\phi(s')$, then you would get the actor-critic setup, where we have an actor (the policy network) that selects actions to perform the rollout and a critic (the value network) that is used to compute the returns, i.e. it grades the performance. And since you are baseline-ing with the value function you actually get an advantage actor-critic. Note that now you can calculate the return without the episode being finished. Thus, you could step the environment for a few steps, then update the policy, and then continue stepping.
Its all about how you calculate the return.
As a side note: I really don't like the fact that people use "vanilla policy gradient" to mean that they are using a Monte-Carlo estimate for the return. In my opinion "vanilla" policy gradient means that you perform a "vanilla" update of the policy using the calculated gradient, i.e.: $$ \theta_{new} = \theta - \alpha \nabla J(\theta).$$ Instead of a "vanilla" update you could update the weights using the natural gradient (TRPO) or you could perform multiple clipped updates (PPO) (see here for more). There are of course other types of policy gradient algorithms, but the idea is that once you have the gradient estimate you do not perform a simple update in the direction of the gradient, but instead do something more sophisticated with it.
