11
votes
Accepted
Why are lambda returns so rarely used in policy gradients?
That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using $G_t^{\...
- 10.1k
9
votes
Why are lambda returns so rarely used in policy gradients?
Recent actor-critic algorithms do use $\lambda$-returns, but they are disguised as something called the Generalized Advantage Estimator defined as $A^{GAE}_t = \sum_{i=0}^{\infty} (\gamma\lambda)^i \...
- 226
7
votes
Accepted
Can TD($\lambda$) be used with deep reinforcement learning?
Eligibility traces is a method of weighting between temporal-difference "targets" and Monte-Carlo "returns". In practice, for example, instead of using the one-step TD target, $r_t ...
- 39.7k
5
votes
Accepted
What is the intuition behind TD($\lambda$)?
TD($\lambda$) can be thought of as a combination of TD and MC learning, so as to avoid to choose one method or the other and to take advantage of both approaches.
More precisely, TD($\lambda$) is ...
- 39.7k
4
votes
Accepted
Why am I getting the incorrect value of lambda?
$TD(\lambda)$ return has the following form:
\begin{equation}
G_t^\lambda = (1 - \lambda) \sum_{n=1}^{\infty} \lambda^{n-1} G_{t:t+n}
\end{equation}
For you MDP $TD(1)$ looks like this:
\begin{align}
...
- 2,336
4
votes
Accepted
How is $\Delta$ updated in true online TD($\lambda$)?
Let us denote the state we are in at time $t$ by $S_t$. Then at iteration $t$ we create a placeholder $V_{old} = V(S_{t+1})$ for the state we will transition into. We then update the value function $V(...
- 4,685
2
votes
Accepted
Why not more TD(𝜆) in actor-critic algorithms?
Theoretically, nothing precludes the use of $\lambda$-returns in actor-critic methods. The $\lambda$-return is an unbiased estimator of the Monte Carlo (MC) return, which means they are essentially ...
- 226
2
votes
What is the intuition behind TD($\lambda$)?
I am a novice in Reinforcement Learning and I have been struggling for several monthes about the TD()'s logic. Initially it seemed to me that it was a successfull purely heuristic formula without any ...
- 221
1
vote
When using TD(λ), how do you calculate the eligibility trace per input & weight of a neural network neuron?
Eligibility traces are not very common in deep-RL, but I suppose that if your network is not huge they might work. I think you are describing the backward view of eligibility traces since you are ...
- 11
1
vote
Why am I getting the incorrect value of lambda?
The previous answer from Brale is mostly correct but is missing a large detail to get the precise answer.
Given this is a question from a GT course homework, I only want to leave pointers so those ...
- 11
Only top scored, non community-wiki answers of a minimum length are eligible