# Questions tagged [temporal-difference-methods]

For questions related to the temporal-difference reinforcement learning (RL) algorithms, which is a class of model-free (that is, they do not use the transition and reward function of the MDP) RL algorithms which learn by bootstrapping from the current estimate of the value function (that is, they use one estimate to update another estimate).

90 questions
Filter by
Sorted by
Tagged with
23 views

I'm reading Sutton's "Learning to Predict by the Methods of Temporal Differences" and I'm getting hung up on a derivation (p. 14). We are considering (observation-sequence, outcome) pairs. [... 1 vote 1 answer 21 views ### Why is importance sampling ratio in n-step TD multiplying error rather than return n-step return? Why is importance sampling ratio in n-step TD is multiplying error rather than return? In Monte Carlo methods for state values, importance sampling ratio was simply a multiplier for the return. • 831 3 votes 0 answers 83 views ### Sutton & Barto: Exercise 7.11 mistake? Exercise from the book: 7.11 Show that if the approximate action values are unchanging, then the tree-backup return (7.16) can be written as a sum of expectation-based TD errors: \begin{align*} &... • 143 1 vote 1 answer 41 views ### Unclear sentence in Sutton-Barto in Temporal-Difference chapter Sutton-Barto, Chapter 6, page:130 By 8000 time steps, the greedy policy was long since optimal (a trajectory from it is shown inset); continued \epsilon-greedy exploration kept the average episode ... • 831 0 votes 0 answers 25 views ### Derivation of update rule for Off-policy TD(0) with importance sampling ratio Sutton-Barto, second edt, page 128, Exercise 6.7: Design an off-policy version of the TD(0) update that can be used with arbitrary target policy \pi and covering behavior policy b, using at each ... • 831 0 votes 0 answers 19 views ### Pseudocode for batch TD(0) This is from Sutton-Barto, second edt, page 126: Suppose there is available only a finite amount of experience, say 10 episodes or 100 time steps. In this case, a common approach with incremental ... • 831 1 vote 0 answers 78 views ### Calculating TD(\lambda) returns, reinforcement learning I am having some trouble with answering the following question: A rat is involved in an experiment. It experiences one episode. At the first step it hears a bell. At the second step it sees a light. ... • 13 2 votes 1 answer 231 views ### Where does the TD formula for tic-tac-toe in Sutton & Barto come from? In section 1.5 of the book "Reinforcement Learning: An Introduction" by Sutton and Barto they use tic-tac-toe as an example of an RL use case. They provide the following temporal ... • 167 2 votes 2 answers 358 views ### Is the Bellman backup unbiased? This is comes from cs285 2023Fall hw3. In my opinion, if \hat{Q} is unbiased estimate of Q, then \begin{align} \mathbb{E}_{D \sim P}[B_{D}\hat{Q} - B_{D}Q] &= \mathbb{E}_{D \sim P}[r(s,a) +... 0 votes 0 answers 25 views ### Derivation of TD(0) with Gradient Correction (TDC) I'm trying to understand the derivation of TDC from Sutton and Barto, but I'm stuck on the steps between lines 2 and 3. It seems that the importance sampling ratio just "disappears" between ... 3 votes 1 answer 210 views ### Convergence of epsilon greedy policy (with no epsilon decay) using TD Learning? If I create a policy using the q-values of an epsilon greedy policy using the Sarsa algorithm (not changing the epsilon with each episode), will it converge to the optimal solution to the MDP? I am ... 0 votes 1 answer 84 views ### Why does my implementation of TD(0) not work? I am trying to implement TD(0) among other RL Policy Evaluation techniques. I have also implemented the dynamic programming approach for a given model of the world and FV Monte Carlo and EV Monte ... • 33 0 votes 0 answers 62 views ### Finding a value at which TD(lambda) and Monte Carlo Are Similar Let's say that I'm trying to find a value of lambda for which running a TD(lambda) method will produce the same results as a Monte Carlo method (within a small margin or error, of course). One of the ... 0 votes 0 answers 31 views ### Correction of value update in off policy TD(0) In this question (inherited from slides coming from D. Silver) they are arguing that the value update in off policy TD(0) should be: $$V(S_t) = V(S_t) + \alpha \left( \frac{ \pi(A_t|S_t)}{\mu (A_t|... • 2,283 0 votes 1 answer 108 views ### How to calculate TD error? I just start to learn reinforcement learning and confuse with TD error. We calculate temporal difference with V(t) = V(t) + α[Rt+1 + γV(t+1) - V(t)] where inside bracket is called TD error. The ... • 101 1 vote 1 answer 80 views ### TD Leaf value function update I'm currently watching a RL course by David Silver and he explains the update of TD Leaf, here is the slide: He says that if, at the next turn (after we played red and the opponent played blue) ... 0 votes 1 answer 75 views ### Consequence of Dvoretzky Stochastic Approximation Theorem I am trying to understand all the steps to prove the TD0 algorithm, and I am following a proof which uses a theorem of Tommi Jaakkola, Michael I. Jordan and Satinder P. Singh, in the paper: On the ... • 19 0 votes 1 answer 43 views ### Time index in TD(0) return in TD(0). for the return we have: G_{t:t+1}=r_{t+1}+\gamma v_t(s_{t+1}). Why is the time index on right hand side in v is t? • 831 1 vote 1 answer 46 views ### Confusion in subscript for n-step TD(0) This is n-step TD(0) update rule: v_{k+n}(x_k)=v_{k+n-1}(x_k)+\alpha [g_{k:k+n}-v_{k+n-1}(x_k)] Why is the subscript on the left hand-side of equation "k+n", not "k+n-1"? Does ... • 831 1 vote 1 answer 185 views ### Where is the problem: in batch TD(0) algorithm or in the code to solve AB problem in Sutton-Barto RL book? Here is the batch TD(0) algorithm: Here is the AB example I want to solve using batch TD(0): And finally here is my Matlab code: % eps1: A 0 B 0 % eps2: B 1 % eps3: B 1 % eps4: B 1 % eps5: B 1 % ... • 831 1 vote 1 answer 405 views ### Proof of convergence of TD(0) algorithm I am looking for a proof of the following tabular TD(0) algorithm: However, I can only find proofs with the more general TD(\lambda) algorithm and I am having problems understanding them. In ... • 19 0 votes 1 answer 92 views ### How can I get Q-Learning (1 step off policy) update from n-step off policy learning update? In Sutton and Barto we have expressions for Q-Learning and n-step Off policy learning. The former ought to be the 1-step limit of the latter but I cannot see it working out that way. What am I missing?... 1 vote 1 answer 266 views ### Using TD algorithms, if the value function of terminal states is always 0, why would a policy ever choose it? Temporal difference algorithms (TD(\lambda)) are tabular solutions to reinforcement learning problems. That is, they create a table of all the states in the problem, and estimate the expected long-... • 148 1 vote 2 answers 64 views ### Given a set of trajectories produced by a fixed policy, what is the the standard approach to estimate Q? Let's say that I have a set of trajectories \mathcal{D} = \{\tau_1, \dots, \tau_n\} produced by an agent acting in a (episodic) MDP with a fixed policy \pi. I would like to estimate the Q ... • 183 1 vote 1 answer 115 views ### If Least-Squares TD is computationally more expensive, then why is it more data efficient than semi-gradient TD(0)? In Sutton-Barto (Section: 9.8 Least-Squares TD, page 228): Authors say that Least-Squares TD is the most "data efficient" form of linear TD(0). Later, in this section, they say the ... • 199 0 votes 1 answer 172 views ### When using TD(λ), how do you calculate the eligibility trace per input & weight of a neural network neuron? I have a Neural Network, each Neuron is made up of inputs, weights, and output. I have potentially multiple hidden layers. The activation function executed against the output is not known by the ... • 220 1 vote 0 answers 111 views ### What is 'eligibility' in intuitive terms in TD(\lambda) learning? I am watching the lecture from Brown University (in udemy) and I am in the portion of Temporal Difference Learning. In the pseudocode/algorithm of TD(1) (seen in the screenshot below), we initialise ... • 185 0 votes 1 answer 234 views ### Why t=τ+n-1 instead of t=τ+n in n-step TD? If \tau is the time, whose state’s estimate is being updated, and t is the current time, then, in n-step TD method, we have t=\tau+n (because we have to wait n-steps, before we can update). ... • 831 2 votes 1 answer 71 views ### Why do we have t as subscript in V instead of t+1 in the expression of G_{t:t+1}? In one-step TD updates, the target is the first reward plus the discounted estimated value of the next state, which we call the one-step return (page 143 of Sutton & Barto):$$ G_{t:t+1} \... • 831 5 votes 1 answer 199 views ### In TD(0) with linear function approximation, why is the gradient of\hat v(S^{\prime}, \mathbf w)$wrt parameters$\mathbf w\$ not considered?

I am reading these slides. On page 38, the update for the parameters for the linear function approximation of TD(0) is given. I have a doubt regarding this. The cost function (RMSE) is given on page ...
• 53
502 views

• 113
205 views

### How should I implement the state transition when it is a Gaussian distribution?

I am reading this paper Anxiety, Avoidance and Sequential Evaluation and is confused about the implementation of a specific lab study. Namely, the authors model what is called the Balloon task using a ...
• 182
471 views

### When calculating the cost in deep Q-learning, do we use both the input and target states?

I just finished Andrew Ngs's deep learning specialization, but RL was not covered, so I don't know the basics of RL. So, I have been having trouble understanding the cost function in deep Q-learning. ...
• 323
215 views

### Is the expected value we sample in TD-learning action-value Q or state-value V?

Both MC and TD are model-free and they both follow a sample trajectory (in the case of TD, the trajectory is cut-short) to estimate the return (we basically are sampling Q values). Other than that, ...
• 155
2k views

### How to determine if Q-learning has converged in practice？

I am using Q-learning and SARSA to solve a problem. The agent learns to go from the start to the goal without falling in the holes. At each state, I can choose the action corresponding to the maximum ...
318 views

• 693