11 votes
Accepted

Why are lambda returns so rarely used in policy gradients?

That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using $G_t^{\...
user avatar
  • 9,379
9 votes

Why are lambda returns so rarely used in policy gradients?

Recent actor-critic algorithms do use $\lambda$-returns, but they are disguised as something called the Generalized Advantage Estimator defined as $A^{GAE}_t = \sum_{i=0}^{\infty} (\gamma\lambda)^i \...
user avatar
7 votes
Accepted

Can TD($\lambda$) be used with deep reinforcement learning?

Eligibility traces is a method of weighting between temporal-difference "targets" and Monte-Carlo "returns". In practice, for example, instead of using the one-step TD target, $r_t ...
user avatar
  • 33.8k
4 votes
Accepted

Why am I getting the incorrect value of lambda?

$TD(\lambda)$ return has the following form: \begin{equation} G_t^\lambda = (1 - \lambda) \sum_{n=1}^{\infty} \lambda^{n-1} G_{t:t+n} \end{equation} For you MDP $TD(1)$ looks like this: \begin{align} ...
user avatar
  • 2,226
4 votes
Accepted

How is $\Delta$ updated in true online TD($\lambda$)?

Let us denote the state we are in at time $t$ by $S_t$. Then at iteration $t$ we create a placeholder $V_{old} = V(S_{t+1})$ for the state we will transition into. We then update the value function $V(...
user avatar
3 votes
Accepted

What is the intuition behind TD($\lambda$)?

TD($\lambda$) can be thought of as a combination of TD and MC learning, so as to avoid to choose one method or the other and to take advantage of both approaches. More precisely, TD($\lambda$) is ...
user avatar
  • 33.8k
2 votes
Accepted

Why not more TD(𝜆) in actor-critic algorithms?

Theoretically, nothing precludes the use of $\lambda$-returns in actor-critic methods. The $\lambda$-return is an unbiased estimator of the Monte Carlo (MC) return, which means they are essentially ...
user avatar
1 vote

When using TD(λ), how do you calculate the eligibility trace per input & weight of a neural network neuron?

Eligibility traces are not very common in deep-RL, but I suppose that if your network is not huge they might work. I think you are describing the backward view of eligibility traces since you are ...
user avatar
  • 11
1 vote

What is the intuition behind TD($\lambda$)?

I am a novice in Reinforcement Learning and I have been struggling for several monthes about the TD()'s logic. Initially it seemed to me that it was a successfull purely heuristic formula without any ...
user avatar
1 vote

Why am I getting the incorrect value of lambda?

The previous answer from Brale is mostly correct but is missing a large detail to get the precise answer. Given this is a question from a GT course homework, I only want to leave pointers so those ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible