Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

# Questions tagged [eligibility-traces]

For questions related to the reinforcement learning technique called "eligibility traces", which combines temporal-difference and Monte Carlo methods.

14 questions
Filter by
Sorted by
Tagged with
1 vote
28 views

### What is 'eligibility' in intuitive terms in TD($\lambda$) learning?

I am watching the lecture from Brown University (in udemy) and I am in the portion of Temporal Difference Learning. In the pseudocode/algorithm of TD(1) (seen in the screenshot below), we initialise ...
• 175
63 views

### Watkins' Q(λ) with function approximation: why is gradient not considered when updating eligibility traces for the exploitation phase?

I'm implementing the Watkins' Q(λ) algorithm with function approximation (in 2nd edition of Sutton & Barto). I am very confused about updating the eligibility traces because, at the beginning of ...
81 views

The pseudocode below is taken from Barto and Sutton's "Reinforcement Learning: an introduction". It shows an actor-critic implementation with eligibility traces. My question is: if I set \... 2 votes 1 answer 110 views ### How to prove the formula of eligibility traces operator in reinforcement learning? I don't understand how the formula in the red circle is derived. The screenshot is taken from this paper • 23 3 votes 0 answers 47 views ### Why weighting by lambda that sums to 1 ensures convergence in eligibility trace? In Sutton and Barto's Book in chapter 12, they state that if weights sum to 1, then an equation's updates have "guaranteed convergence properties". Actually why it ensures convergence? There ... 2 votes 1 answer 116 views ### How do I derive the gradient with respect to the parameters of the softmax policy? The gradient of the softmax eligibility trace is given by the following: \begin{align} \nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\ &= \phi(s,a) - \sum_{... 1 vote 1 answer 61 views ### Applying Eligibility Traces to Q-Learning algorithm does not improve results (And might not function well) I am trying to apply Eligibility Traces to a currently working Q-Learning algorithm. The reference code for the Q-Learning algorithm was taken from this great blog ... 5 votes 1 answer 463 views ### Why not more TD(𝜆) in actor-critic algorithms? Is there either an empirical or theoretical reason that actor-critic algorithms with eligibility traces have not been more fully explored? I was hoping to find a paper or implementation or both for ... • 115 1 vote 0 answers 38 views ### How is the general return-based off-policy equation derived? I'm wondering how is the general return-based off-policy equation in Safe and efficient off-policy reinforcement learning derived $$\mathcal{R} Q(x, a):=Q(x, a)+\mathbb{E}_{\mu}\left[\sum_{t \geq 0} \... • 247 1 vote 1 answer 799 views ### How can the \lambda-return be defined recursively? The \lambda-return is defined as$$G_t^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t:t+n}$$where$$G_{t:t+n} = R_{t+1}+\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n})... • 1,928 4 votes 1 answer 80 views ### How to apply or extend theQ(\lambda)$algorithm to semi-MDPs? I want to model an SMDP such that time is discretized and the transition time between the two states follows an exponential distribution and there would be no reward between the transition. Can I ... • 451 6 votes 1 answer 2k views ### Can TD($\lambda\$) be used with deep reinforcement learning?

TD lambda is a way to interpolate between TD(0) - bootstrapping over a single step, and, TD(max), bootstrapping over the entire episode length, or, Monte Carlo. Reading the link above, I see that an ...
• 669
1 vote