1
$\begingroup$

In general, the advantage function is defined as:

$A^\pi\left(s_t, a_t\right):=Q^\pi\left(s_t, a_t\right)-V^\pi\left(s_t\right)$

So far, I understand this formular like this:

With the advantage function, we can compare, how good (positive sign, i.e. $A^\pi > 0$) or bad (negative sign , i.e. $A^\pi < 0$) a specific action $a$ in state $s$ is compared against the average of all actions at time step $t$.

However, in the paper High-Dimensional Continuous Control Using Generalized Advantage Estimation (https://arxiv.org/pdf/1506.02438.pdf), they introduce the following notation:

$E_{s_{0: \infty} \\ a_{0: \infty}}\left[\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right) \nabla_\theta \log \pi_\theta(a_t \mid s_t)\right]$ (see equation $7$)

I am a little bit confused by the term

$\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$

What does this expression mean? How can we estimate the advantage at time step $t$, using all actions and states starting from the starting state?

$\endgroup$

1 Answer 1

1
$\begingroup$

In policy gradient methods reducing bias is crucial to obtain more accurate gradient estimates for updating the policy in a stochastic gradient ascent fashion. The usual advantage function as you expressed only implicitly addresses credit assignment for the current state and the specific on-policy's action at time step $t$, while the generalized advantage estimation (GAE) expressed by equation 16 explicitly addresses credit assignment by capturing the temporal consistency of advantages and considering the cumulative effects of the on-policy's actions from the initial state to the current state and beyond until the end of its episode during each parameters update time step $t$. Also from equation 16 you see GAE has a flexible bias-variance trade-off parameter $λ$ to further fine tune for each problem, and even if it's biased when it's not $γ$-just, the bias must be very small as reflected in equation 14.

Therefore it's clear that the advantage estimator $\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$ referenced in equation 7 is a function of the entire trajectory of an episode in general (the 2 semicolons mean the independent variables could be all states and their on-policy actions of any episode), and if it's of the form expressed in Proposition 1 then it's $γ$-just. The usual discounted advantage function is a special $γ$-just case of the advantage estimator $\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$ as mentioned immediately after Proposition 1, and the GAE expressed in equation 16 is the generalized case of $\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$ considering the on-policy's advantage over the entire episode to become completely unbiased when $λ=1$ in theory.

$\endgroup$
3
  • $\begingroup$ I think you mean that in policy gradient methods we are interested in reducing variance, not bias? Since with this method explained in the paper, we are introducing bias, we are not reducing it. $\endgroup$
    – kklaw
    Commented Feb 18 at 8:13
  • $\begingroup$ See equation 18 and the explanation after it: GAE(γ, 1) is γ-just regardless of the accuracy of V, but it has high variance due to the sum of terms. So when λ=1 GAE is unbiased estimator for γ-discounted advantage function. But as the footnote on page 3 notes that here we are concerned with obtaining an unbiased estimate of $g^γ$, which is a biased estimate of the policy gradient of the undiscounted MDP. PG method has to take discounted and undiscounted MDP separately as one of its complications compared to value based methods. $\endgroup$
    – cinch
    Commented Feb 19 at 0:21
  • $\begingroup$ Anyway the flexible bias-variance trade-off capability of GAE is its feature compared to the usual advantage function which typically has low variance but high bias which is also mentioned in the same paper and this has little to do with your question about $\hat{A}_t\left(s_{0: \infty}, a_{0: \infty}\right)$, and equation 16 is the constructive way to estimate the generalized advantage at time step $t$ using all actions and states starting from the starting state as you requested. $\endgroup$
    – cinch
    Commented Feb 19 at 0:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .