Questions tagged [bellman-operators]

For questions related to Bellman operators in the context of reinforcement learning.

Filter by
Sorted by
Tagged with
2 votes
0 answers
44 views

How the proof of the contraction of variance for distributional Bellman operator follows

I am stuck at the proof of the contraction of variance for distributional Bellman operator from the paper, in which it is defined as and the proof is stated as In its second part, how is the ...
0 votes
0 answers
28 views

How to prove the offset property in reinforcement learning? [closed]

Who can help me to prove the offset property in reinforcement learning for $Q$ function: $T(Q(s, a)+ce)=TQ(s,a)+ \gamma ce$ $T_\pi(Q(s, a)+ce)=T_\pi Q(s,a)+ \gamma ce$ where $T$ is bellman operator, $\...
  • 1
2 votes
1 answer
175 views

Are my proofs that the Bellman operators are contractions correct?

Introduction I'm studying Reinforcement Learning, and in order to increase my understanding I've been challenging myself by trying to write proofs that show that the right hand side of the Bellman ...
  • 23
3 votes
1 answer
1k views

What do the terms 'Bellman backup' and 'Bellman error' mean?

Some RL literature use terms such as: 'Bellman backup' and 'Bellman error'. What do these terms refer to?
1 vote
0 answers
162 views

Off-policy Bellman Operators: Writing Operator and Weight Update Function for a 2-State System

I am studying for RL on my own and was trying to solve this question I came across. Write an operator function $T(w, \pi, \mu, l, g)$ that takes weights $w$, a target policy $\pi$, a behaviour policy ...
1 vote
1 answer
377 views

How to derive matrix form of the Bellman operators?

Reading the Retrace paper (Safe and efficient off-policy reinforcement learning) I saw they often use a matrix form of the Bellman operators, for example as in the picture below. How do we derive ...
8 votes
2 answers
2k views

Why are the Bellman operators contractions?

In these slides, it is written \begin{align} \left\|T^{\pi} V-T^{\pi} U\right\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \tag{9} \label{9} \\ \|T V-T U\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \...
  • 181
26 votes
1 answer
12k views

What is the Bellman operator in reinforcement learning?

In mathematics, the word operator can refer to several distinct but related concepts. An operator can be defined as a function between two vector spaces, it can be defined as a function where the ...
  • 36.4k