3
$\begingroup$

Why does A2C use the actual returns from an episode in calculating the advantage instead of using a bellman equation style estimate of the value?

Basically, why this:

$A(s,a) = \sum_t\gamma^tr_t - V(s)$

instead of

$A(s,a) = r + \gamma V(s') - V(s)$

This blog post seems to describe it one way, but their code is the other way: https://towardsdatascience.com/understanding-actor-critic-methods-931b97b6df3f

I feel there's some big gap in my understanding, but would appreciate some insight.

$\endgroup$

1 Answer 1

2
+50
$\begingroup$

Let's look at how they derived the equation for that. Recall that $A(s,a) = Q(s,a) - V(s)$, so in the first equation, it is $Q(s,a) = \sum_t \gamma^tr_t$. We should noted that, the original Policy Gradient uses $A(s,a) = \sum_t \gamma^tr_t - b$, where $b$ is any baseline.

The first equation $A(s,a) = \sum_t \gamma^t r_t - b$, the original Policy Gradient, is unbiased because it is a single-sample estimate, but it has high variance because it sums all rewards, thus requires a good learning rate.

The second equation $A(s,a) = r + \gamma V(s') - V(s)$ has the advantage that it dramatically lowers the variance since it relies on all possibilities that can occur (Policy Gradient has the term "sum of the Advantage since step 0"). However, it is not unbiased because if the approximation of $V$ is incorrect, the policy gradient will fail to converge. Now this happens regularly with Neural Networks (NN) in Deep RL, because random initialization of NN already makes incorrect approximation of $V$.

In the end, what we want is an unbiased Advantage estimator, but needs to have low variance as well. We then use a critic as a the state-dependent baselines $V(s)$ for the Advantage calculation, that is the equation $A(s,a) = \sum_t \gamma^t r_t - V(s)$ you see in A2C.

You can read more details in the Spinning Up docs, RAIL Lecture 6, and RAIL homework 2 if you want to know why they come up with such equation.

$\endgroup$
1
  • 2
    $\begingroup$ Another way to balance the bias-variance trade-off is to use the Generalized Advantage Estimation (arxiv.org/abs/1506.02438) $\endgroup$
    – entropy07
    Jan 29, 2023 at 7:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .