Questions tagged [temporal-difference]

For questions related to the temporal-difference reinforcement learning (RL) algorithms, which is a class of model-free (that is, they do not use the transition and reward function of the MDP) RL algorithms which learn by bootstrapping from the current estimate of the value function (that is, they use one estimate to update another estimate).

Filter by
Sorted by
Tagged with
0
votes
0answers
16 views

How is the general return-based off-policy equation derived?

I'm wondering how is the general return-based off-policy equation in Safe and efficient off-policy reinforcement learning derived $$\mathcal{R} Q(x, a):=Q(x, a)+\mathbb{E}_{\mu}\left[\sum_{t \geq 0} \...
3
votes
1answer
45 views

Convergence of Semi gradient TD(0) with non-linear function approximation

I am looking for a result that shows convergence of semi-gradient TD(0) algorithm with non-linear function approximation for on-policy prediction. Specifically, the update equation is given by (...
3
votes
1answer
33 views

Confusion about temporal difference learning

I have a couple of small questions about the David Silver lecture about reinforcement learning, lecture slides (slides 23, 24). More specifically it is about the temporal difference algorithm: $$V(...
3
votes
1answer
101 views

How to show temporal difference methods converge to MLE?

In chapter 6 of Sutton and Barto (p. 128), they claim temporal difference converges to the maximum likelihood estimate (MLE). How can this be shown formally?
1
vote
1answer
54 views

Understanding TD(0) algorithm implementation

I came across the $TD(0)$ algorithm from Sutton and Barto: Clearly, the only difference of TD methods with the MC methods is that TD method is not waiting till the end of the episode to update the $...
1
vote
0answers
28 views

How do I know if the assumption of a static environment is made?

An important property of a reinforcement learning problem is whether the environment of the agent is static, which means that nothing changes if the agent remains inactive. Different learning methods ...
5
votes
1answer
280 views

Understanding the equation of TD(0) in the paper “Learning to predict by the methods of temporal differences”

In the paper Learning to predict by the methods of temporal differences (p. 15), the weights in the temporal difference learning are updated as given by the equation $$ \Delta w_t = \alpha \left(P_{t+...
2
votes
0answers
69 views

Infinite horizon in Reinforcement Learning

I read this article: "Towards Autonomous Data Ferry Route Design through Reinforcement Learning" by Daniel Henkel and Timothy X Brown. It specifies an infinite horizon problem where they use as a ...
5
votes
2answers
221 views

Why am I getting the incorrect value of lambda?

I am trying to solve for lambda using Temporal Difference Learning I am trying to figure out what lambda I need, to make TD(λ)=TD(1) but I get the incorrect value of lambda. Here is how I did: <...
1
vote
0answers
23 views

Why should we use TD prediction as opposed to TD control algorithms?

Consider a problem where we have a finite number of states and actions. Given a state and an action, we can easily know the reward and the next state (deterministic). The state space is really large ...
2
votes
1answer
69 views

What is the relation between Monte Carlo and model-free algorithms?

Monte Carlo (MC) methods are methods that use some form of randomness or sampling. For example, we can use an MC method to approximate the area of a circle inside a square: we generate random 2D ...
1
vote
1answer
135 views

How fast does Monte Carlo tree search converge?

How fast does Monte Carlo Tree Search converge? Is there proof that it does converge? How does it compare to Temporal Difference learning in terms of convergence speed (assuming the evaluation step ...
3
votes
1answer
90 views

Understanding the n-step off-policy SARSA update

In Sutton & Barto's book (2nd ed) page 149, there is the equation 7.11 I am having a hard time understanding this equation. I would have thought that we should be moving $Q$ towards $G$, where $...
1
vote
1answer
48 views

On-policy distribution for Emphatic TD

The book by Sutton and Barto discussed in section 11.8 that the convergence of off-policy TD function approximation can be improved by correcting for distribution of states encountered. The section ...
4
votes
1answer
84 views

What are temporal-difference and Monte Carlo methods intuitively?

Intuitively, how do temporal-difference and Monte Carlo methods work in reinforcement learning? How can they be used to solve the reinforcement learning problem?