5
$\begingroup$

Is there either an empirical or theoretical reason that actor-critic algorithms with eligibility traces have not been more fully explored? I was hoping to find a paper or implementation or both for continuous tasks (not episodic) in continuous state-action spaces.

This has been the only related question on SE-AI that I have been able to find Why are lambda returns so rarely used in policy gradients?.

Although I appreciated the dialog and found it useful, I was wondering if there was any further detail or reasoning that might help explain the void.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Theoretically, nothing precludes the use of $\lambda$-returns in actor-critic methods. The $\lambda$-return is an unbiased estimator of the Monte Carlo (MC) return, which means they are essentially interchangeable. In fact, as discussed in High-Dimensional Continuous Control Using Generalized Advantage Estimation, using the $\lambda$-return instead of the MC return can actually help reduce the variance of gradient updates.

The above is similar to my answer in the other question you linked, so let me try to answer your question more specifically. Even though we can use $\lambda$-returns, why are they not too common in practice? I suspect there might be a few reasons:

  1. Empirically, faster credit assignment might be more desirable than lower variance. Sometimes the learning speed of your algorithm is constrained simply by how quickly you can learn about the consequences of certain actions. In this case, it is faster to use the MC return, even if it theoretically has higher variance than the $\lambda$-return.

  2. When proposing a new algorithm, adding $\lambda$-returns to it might give it an "unfair" advantage if the other baseline methods do not use them (the reviewers would not like this), so researchers tend to favor simpler 1-step or MC returns for the sake of consistency. I would guess this is why you don't typically see $\lambda$-returns in papers that propose new actor-critic methods. In some sense, it is generally assumed that you could always add $\lambda$-returns to them later and probably get better performance.$^*$

  3. A decent number of deep RL researchers don't know what $\lambda$-returns are. This is especially true if they come from a pure deep learning background; they may have never read Reinforcement Learning: An Introduction which is where most people are introduced to TD($\lambda$) and $\lambda$-returns.

$^*$Exceptions to this are papers like mine (Reconciling $\lambda$-Returns with Experience Replay) where the contribution is the use of $\lambda$-returns in methods that previously could not use them. But actor-critic methods are pretty straightforward to combine with $\lambda$-returns.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ Why are $\lambda$ returns unbiased estimators of MC returns, given that they are an average of returns that go from TD(0) to TD(1)? Maybe this could be a question on the main site. Anyway, you say that $\lambda$ returns are unbiased estimators of MC returns, but then you say "In fact, they help to reduce the variance", while the use of MC returns usually leads to high variance. In other words, the "In fact" isn't probably the best way of starting that phrase. Also, I think you should better distinguish between "faster credit assignment" and "faster implementation". $\endgroup$
    – nbro
    Feb 20, 2020 at 12:04
  • 2
    $\begingroup$ The $\lambda$-return is a statistically unbiased estimator of the MC return because it's a weighted average of $n$-step returns, each of which are themselves unbiased estimators of the MC return on average. This should not be confused with the "bias" due to bootstrapping; the terminology is a bit confusing because there are two notions of bias here. I am referring to the former notion: each $n$-step return, in expectation, is equal to the MC return and therefore so is the $\lambda$-return. $\endgroup$
    – Brett Daley
    Feb 20, 2020 at 20:56
  • 2
    $\begingroup$ This also explains why the $\lambda$-return should reduce variance. Each $n$-step return has some variance, but they all have the same expectation. The $\lambda$-return is a convex combination of the $n$-step returns, where the weights are positive and sum to 1. Therefore, the $\lambda$-return must also have the same expectation. But from the variance formula $Var(aX+bY)=a^2 Var(X)+b^2 Var(Y)$ (assuming independence), you can see how this should reduce the variance because $a^2 + b^2 < 1$. This can be viewed as an ensemble method of sorts. $\endgroup$
    – Brett Daley
    Feb 20, 2020 at 21:08
  • 2
    $\begingroup$ Finally, I did intend to say "faster credit assignment". The MC return propagates reward information back in time to all states in the episode. The $\lambda$-return does too, technically/mathematically, but the exponential decay means that it practically stops updating states after some finite number of timesteps. $\endgroup$
    – Brett Daley
    Feb 20, 2020 at 21:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .