Questions tagged [bellman-operators]
For questions related to Bellman operators in the context of reinforcement learning.
8
questions
2
votes
0
answers
44
views
How the proof of the contraction of variance for distributional Bellman operator follows
I am stuck at the proof of the contraction of variance for distributional Bellman operator from the paper, in which it is defined as
and the proof is stated as
In its second part, how is the ...
- 41
0
votes
0
answers
28
views
How to prove the offset property in reinforcement learning? [closed]
Who can help me to prove the offset property in reinforcement learning for $Q$ function:
$T(Q(s, a)+ce)=TQ(s,a)+ \gamma ce$
$T_\pi(Q(s, a)+ce)=T_\pi Q(s,a)+ \gamma ce$
where $T$ is bellman operator, $\...
- 1
2
votes
1
answer
175
views
Are my proofs that the Bellman operators are contractions correct?
Introduction
I'm studying Reinforcement Learning, and in order to increase my understanding I've been challenging myself by trying to write proofs that show that the right hand side of the Bellman ...
- 23
3
votes
1
answer
1k
views
What do the terms 'Bellman backup' and 'Bellman error' mean?
Some RL literature use terms such as: 'Bellman backup' and 'Bellman error'. What do these terms refer to?
- 349
1
vote
0
answers
162
views
Off-policy Bellman Operators: Writing Operator and Weight Update Function for a 2-State System
I am studying for RL on my own and was trying to solve this question I came across.
Write an operator function $T(w, \pi, \mu, l, g)$ that takes weights $w$, a target policy $\pi$, a behaviour policy ...
1
vote
1
answer
377
views
How to derive matrix form of the Bellman operators?
Reading the Retrace paper (Safe and efficient off-policy reinforcement learning) I saw they often use a matrix form of the Bellman operators, for example as in the picture below. How do we derive ...
- 223
8
votes
2
answers
2k
views
Why are the Bellman operators contractions?
In these slides, it is written
\begin{align}
\left\|T^{\pi} V-T^{\pi} U\right\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \tag{9} \label{9} \\
\|T V-T U\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \...
- 181
26
votes
1
answer
12k
views
What is the Bellman operator in reinforcement learning?
In mathematics, the word operator can refer to several distinct but related concepts. An operator can be defined as a function between two vector spaces, it can be defined as a function where the ...
- 36.4k