5
$\begingroup$

I've seen monte-carlo reward $G_{t}$ used in REINFORCE and TD($0$) reward $r_t + \gamma Q(s', a')$ used in vanilla actor-critic. I've never seen someone use lambda reward $G^{\lambda}_{t}$ in these situations, nor in any other algorithms.

Is there a specific reason for this? Could there be performance improvements if we used $G^{\lambda}_{t}$?

$\endgroup$
7
$\begingroup$

That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using $G_t^{\lambda}$ returns for the critic (value function estimator), but also for the actor's policy gradients.

Note that you do not explicitly see the $G_t^{\lambda}$ returns mentioned in the pseudocode. They are being used implicitly through eligibility traces, which allow for an efficient online implementation (the "backward view").


I do indeed have the impression that such uses are fairly rare in recent research though. I haven't personally played around with policy gradient methods to tell from personal experience why that would be. My guess would be that it is because policy gradient methods are almost always combined with Deep Neural Networks, and variance is already a big enough problem in training these things without starting to involve long-trajectory returns.

If you use large $\lambda$ with $\lambda$-returns, you get low bias, but high variance. For $\lambda = 1$, you basically get REINFORCE again, which isn't really used much in practice, and has very high variance. For $\lambda = 0$, you just get one-step returns again. Higher values for $\lambda$ (such as $\lambda = 0.8$) tend to work very well in my experience with tabular methods or linear function approximation, but I suspect the variance may simply be too much when using DNNs.

Note that it is quite popular to use $n$-step returns with a fixed, generally fairly small, $n$ in Deep RL approaches. For instance, I believe the original A3C paper used $5$-step returns, and Rainbow uses $3$-step returns. These often work better in practice than $1$-step returns, but still have reasonably low variance due to using small $n$.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thanks for the answer! It makes sense that lambda returns would add more variance compared to the more common TD(small n) rewards, so if variance reduction is a priority then one would use TD(0) or TD(small n). $\endgroup$ – jhinGhin Jan 18 '19 at 21:29
7
$\begingroup$

Recent actor-critic algorithms do use $\lambda$-returns, but they are disguised as something called the Generalized Advantage Estimator defined as $A^{GAE}_t = \sum_{i=0}^{\infty} \gamma^i \delta_{t+i}$ where $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$. This turns out to be identically equal to $[G^\lambda_t - V(s_t)]$, i.e. the $\lambda$-return with a value-function baseline subtracted from it. Theoretically, any actor-critic gradient method could use this quite easily; it was combined with TRPO in the GAE paper, and later used for PPO. Similarly, ACER uses an off-policy variant known as Retrace($\lambda$).

For replay methods like DQN or DDPG, it is harder to implement $\lambda$-returns. This is why they have historically defaulted to $n$-step returns as @DennisSoemers mentioned. I recently published a paper that describes a way to efficiently combine $\lambda$-returns with experience replay, which I hope will increase the popularity of $\lambda$-returns for these methods.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.