# Questions tagged [eligibility-traces]

For questions related to the reinforcement learning technique called "eligibility traces", which combines temporal-difference and Monte Carlo methods.

12 questions
Filter by
Sorted by
Tagged with
59 views

The pseudocode below is taken from Barto and Sutton's "Reinforcement Learning: an introduction". It shows an actor-critic implementation with eligibility traces. My question is: if I set \... 1answer 99 views ### How to prove the formula of eligibility traces operator in reinforcement learning? I don't understand how the formula in the red circle is derived. The screenshot is taken from this paper 0answers 37 views ### Why weighting by lambda that sums to 1 ensures convergence in eligibility trace? In Sutton and Barto's Book in chapter 12, they state that if weights sum to 1, then an equation's updates have "guaranteed convergence properties". Actually why it ensures convergence? There ... 1answer 98 views ### How do I derive the gradient with respect to the parameters of the softmax policy? The gradient of the softmax eligibility trace is given by the following: \begin{align} \nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\ &= \phi(s,a) - \sum_{... 1answer 38 views ### Applying Eligibility Traces to Q-Learning algorithm does not improve results (And might not function well) I am trying to apply Eligibility Traces to a currently working Q-Learning algorithm. The reference code for the Q-Learning algorithm was taken from this great blog ... 1answer 283 views ### Why not more TD(𝜆) in actor-critic algorithms? Is there either an empirical or theoretical reason that actor-critic algorithms with eligibility traces have not been more fully explored? I was hoping to find a paper or implementation or both for ... 0answers 35 views ### How is the general return-based off-policy equation derived? I'm wondering how is the general return-based off-policy equation in Safe and efficient off-policy reinforcement learning derived $$\mathcal{R} Q(x, a):=Q(x, a)+\mathbb{E}_{\mu}\left[\sum_{t \geq 0} \... 1answer 585 views ### How can the \lambda-return be defined recursively? The \lambda-return is defined as$$G_t^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t:t+n}$$where$$G_{t:t+n} = R_{t+1}+\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n})... 1answer 71 views ### How to apply or extend theQ(\lambda)$algorithm to semi-MDPs? I want to model an SMDP such that time is discretized and the transition time between the two states follows an exponential distribution and there would be no reward between the transition. Can I ... 1answer 2k views ### Can TD($\lambda\$) be used with deep reinforcement learning?

TD lambda is a way to interpolate between TD(0) - bootstrapping over a single step, and, TD(max), bootstrapping over the entire episode length, or, Monte Carlo. Reading the link above, I see that an ...