Questions tagged [bellman-equations]
For questions related to the Bellman equations in the context of reinforcement learning (and other artificial intelligence subfields).
70
questions
30
votes
1
answer
17k
views
What is the Bellman operator in reinforcement learning?
In mathematics, the word operator can refer to several distinct but related concepts. An operator can be defined as a function between two vector spaces, it can be defined as a function where the ...
- 40.5k
10
votes
2
answers
4k
views
Why are the Bellman operators contractions?
In these slides, it is written
\begin{align}
\left\|T^{\pi} V-T^{\pi} U\right\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \tag{9} \label{9} \\
\|T V-T U\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \...
- 201
8
votes
2
answers
759
views
Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?
I often see that the state-action value function is expressed as:
$$q_{\pi}(s,a)=\color{red}{\mathbb{E}_{\pi}}[R_{t+1}+\gamma G_{t+1} | S_t=s, A_t = a] = \color{blue}{\mathbb{E}}[R_{t+1}+\gamma v_{\pi}...
- 343
8
votes
2
answers
4k
views
What is the proof that policy evaluation converges to the optimal solution?
Although I know how the algorithm of iterative policy evaluation using dynamic programming works, I am having a hard time realizing how it actually converges.
It appeals to intuition that, with each ...
- 107
8
votes
1
answer
2k
views
How is the DQN loss derived from (or theoretically motivated by) the Bellman equation, and how is it related to the Q-learning update?
I'm doing a project on Reinforcement Learning. I programmed an agent that uses DDQN. There are a lot of tutorials on that, so the code implementation was not that hard.
However, I have problems ...
- 183
8
votes
0
answers
275
views
Is the Bellman equation that uses sampling weighted by the Q values (instead of max) a contraction?
It is proved that the Bellman update is a contraction (1).
Here is the Bellman update that is used for Q-Learning:
$$Q_{t+1}(s, a) = Q_{t}(s, a) + \alpha*(r(s, a, s') + \gamma \max_{a^*} (Q_{t}(s',
...
- 81
7
votes
1
answer
383
views
Why do Bellman equations indirectly create a policy?
I was watching a lecture on policy gradients and Bellman equations. And they say that a Bellman equation indirectly creates a policy, while the policy gradient directly learns a policy. Why is this?
- 693
6
votes
2
answers
2k
views
What is the Bellman Equation actually telling?
What does the Bellman equation actually say? And are there many flavours of that?
I get a little confused when I look for the Bellman equation, because I feel like people are telling slightly ...
- 69
5
votes
1
answer
1k
views
How would I compute the optimal state-action value for a certain state and action?
I am currently trying to learn reinforcement learning and I started with the basic gridworld application. I tried Q-learning with the following parameters:
Learning rate = 0.1
Discount factor = 0.95
...
- 215
4
votes
3
answers
956
views
Why can the Bellman equation be turned into an update rule?
In chapter 4.1 of Sutton's book, the Bellman equation is turned into an update rule by simply changing the indices of it. How is it mathematically justified? I didn't quite get the initiation of why ...
- 123
4
votes
2
answers
144
views
Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?
In equation 3.17 of Sutton and Barto's book:
$$q_*(s, a)=\mathbb{E}[R_{t+1} + \gamma v_*(S_{t+1}) \mid S_t = s, A_t = a]$$
$G_{t+1}$ here have been replaced with $v_*(S_{t+1})$, but no reason has ...
- 147
4
votes
1
answer
169
views
Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?
I was looking at the Bellman equation, and I noticed a difference between the equations used in policy evaluation and value iteration.
In policy evaluation, there was the presence of $\pi(a \mid s)$, ...
- 125
4
votes
1
answer
189
views
In reinforcement learning, does the optimal value correspond to performing the best action in a given state?
I am confused about the definition of the optimal value ($V^*$) and optimal action-value (Q*) in reinforcement learning, so I need some clarification, because some blogs I read on Medium and GitHub ...
- 433
4
votes
1
answer
3k
views
What do the terms 'Bellman backup' and 'Bellman error' mean?
Some RL literature use terms such as: 'Bellman backup' and 'Bellman error'. What do these terms refer to?
- 369
4
votes
1
answer
470
views
How are afterstate value functions mathematically defined?
In this answer, afterstate value functions are mentioned, and that temporal-difference (TD) and Monte Carlo (MC) methods can also use these value functions. Mathematically, how are these value ...
- 40.5k
4
votes
1
answer
923
views
Why we don't use importance sampling in tabular Q-Learning?
Why don't we use an importance sampling ratio in Q-Learning, even though Q-Learning is an off-policy method?
Importance sampling is used to calculate expectation of a random variable by using data ...
- 4,900
4
votes
1
answer
152
views
How to prove the second form of Bellman's equation?
I'd like to prove this "second form" of Bellman's equation: $v(s) = \mathbb{E}[R_{t + 1} + \gamma v(S_{t+1}) \mid S_{t} = s]$ starting from Bellman's equation: $v(s) = \mathbb{E}[G_{t} \mid ...
- 575
4
votes
1
answer
81
views
Could you explain these 2 steps of the derivation of the Bellman equation as a recursive equation in Sutton & Barto?
I am reading the Sutton & Barto (2018) RL textbook.
On page 59, it derives the recursive property of the state-value function as below.
Could you explain the steps of third and fourth equality?
...
- 205
3
votes
1
answer
2k
views
What is the difference between a greedy policy and an optimal policy?
I am struggling to understand what is the difference between an optimal policy and a greedy policy.
Let $F(r_{t+1},s_{t+1}| s_t,a_t)$ be the probability distribution accorting to which, given action $...
- 33
3
votes
2
answers
800
views
Apart from the state and state-action value functions, what are other examples of value functions used in RL?
In reinforcement learning, we often define two functions, the state-value function
$$V^\pi(s) = \mathbb{E}_{\pi} \left[\sum_{k=0}^{\infty}
\gamma^{k}R_{t+k+1} \Bigg| S_t=s \right]$$
and the state-...
- 40.5k
3
votes
1
answer
362
views
Why must the value of a state under an optimal policy equal the expected return for the best action from that state?
The Sutton and Barto reinforcement learning textbook states that
the value of a state under an optimal policy must equal the expected return for the best action from that state.
That is,
$$v_*(s) = \...
3
votes
2
answers
1k
views
How can we find the value function by solving a system of linear equations?
I am following the book "Reinforcement Learning: An Introduction" by Richard Sutton and Andrew Barto, and they give an example of a problem for which the value function can be computed ...
- 143
3
votes
1
answer
95
views
Can we also estimate $V_{\pi}$ with SARSA?
For SARSA, I know we can estimate the action value $Q(s,a)$, and the relationship between $V(s)$ and $Q(s,a)$ is $V_{\pi}(s) = \sum_{a \in \mathcal{A}} \pi(a|s)Q_{\pi}
(s,a)$.
So my question is, can ...
- 31
3
votes
1
answer
347
views
How can we find the value function by solving a system of linear equations without knowing the policy?
An MDP is a Markov Reward Process with decisions, it’s an environment in which all states are Markov. This is what we want to solve. An MDP is a tuple $(S, A, P, R, \gamma)$, where $S$ is our state ...
- 45
3
votes
1
answer
197
views
Why does A2C use the actual returns from an episode in calculating the advantage?
Why does A2C use the actual returns from an episode in calculating the advantage instead of using a bellman equation style estimate of the value?
Basically, why this:
$A(s,a) = \sum_t\gamma^tr_t - V(s)...
- 125
3
votes
2
answers
252
views
How do we get the optimal value-function?
In here it says that: (is it correct?)
$$V^\pi = \sum_{a \in A}\pi(a|s)*Q^\pi(s,a)$$
And we have:
$$ V^*(s) = max_\pi V^\pi(s)$$
Also:
$$ V^*(s) = max_a Q^*(s, a) $$
Can someone demonstrate to me step ...
- 206
3
votes
1
answer
420
views
Can an optimal policy have a value function that has a smaller value for a state than a non-optimal policy?
I'm starting to learn about the Bellman Equation and a question came to my mind.
A policy $\pi$ is optimal if the value $v_\pi(s)$ is greater or equal than the value $v_{\pi'}(s)$ for all states $s \...
- 133
2
votes
2
answers
1k
views
What is the Bellman equation for V(s) in the case of a deterministic environment?
I am currently trying to practice reinforcement learning for an agent on a grid. The grid is deterministic. Since the grid is deterministic, to calculate the value for each grid square from the reward ...
- 145
2
votes
2
answers
234
views
Why is there an expectation sign in the Bellman equation?
In chapter 3.5 of Sutton's book, the value function is defined as:
Can someone give me some clarification about why there is the expectation sign behind the entire equation? Considering that the ...
- 23
2
votes
1
answer
159
views
Bellman equation and inverse matrix method
My problem: why the last formula do not contain information about time $t$?
So if $s^{\prime}=s$, do we have $v_{\pi}(s) = v_{\pi}(s^{\prime})$? But this is not right I guess?
If I am right, that they ...
- 21
2
votes
1
answer
170
views
What is the difference between these two versions of the Bellman equation?
The first version is the one I am most familiar with:
$$V_\pi(s) = \sum_{a}^{}\pi(a|s) \sum_{s'}^{}T(s, a, s')[R(s, a, s') + \gamma V_\pi(s')]$$
where $T(s, a, s')$ represents the probability of ...
- 133
2
votes
1
answer
325
views
Determining to terminate at a reward or not
I am practicing the Bellman equation on Grid world examples and in this scenario, there are numbered grid squares where the agent can choose to terminate and collect the reward equal to the amount ...
- 145
2
votes
1
answer
460
views
calculating the value of a state in an optimal policy analytically and iteratively
I am watching the lecture by Abbeel on MDPs and Reinforcement Learning. The setup of the problem is the classic gridworld with optimal policy (and corresponding values of states) pictured below.
The ...
- 185
2
votes
1
answer
288
views
Where are the parentheses in the Bellman update rule?
I'm not having a lot of intuition about the equation. I have this Bellman update rule:
$$v_{\pi}(s) =\sum_a \pi(a|s)\sum_{s',r} p(s',r|s,a)[r+ \gamma v_{k}(s')]$$
But where are the parenthesis? Is the ...
- 216
2
votes
1
answer
253
views
Are these two definitions of the state-action value function equivalent?
I have been reading the Sutton and Barto textbook and going through David Silvers UCL lecture videos on YouTube and have a question on the equivalence of two forms of the state-action value function ...
- 4,900
2
votes
1
answer
50
views
Proof of existence of optimal policy
I have been trying to prove existence of an optimal policy for RL. I have proved that the Bellman optimality operator, $B: \mathbb{R}^{|\mathcal{S}|} \to \mathbb{R}^{|\mathcal{S}|}$ given by
$$B(v_\pi)...
- 167
2
votes
1
answer
214
views
Proof of bellman optimality equations
I am studying RL and have a hard time proving the existence of an optimal policy. I found some resources online, and I am trying to prove the following theorem:
If there exists a policy $\pi$, state $...
- 167
2
votes
1
answer
734
views
How to derive matrix form of the Bellman operators?
Reading the Retrace paper (Safe and efficient off-policy reinforcement learning) I saw they often use a matrix form of the Bellman operators, for example as in the picture below. How do we derive ...
- 253
2
votes
1
answer
1k
views
Are my proofs that the Bellman operators are contractions correct?
Introduction
I'm studying Reinforcement Learning, and in order to increase my understanding I've been challenging myself by trying to write proofs that show that the right hand side of the Bellman ...
- 53
2
votes
1
answer
2k
views
How to avoid being stuck local optima in q-learning and q-network
When using the Bellman equation to update q-table or train q-network to fit greedy max values, the q-values very often get to the local optima and get stuck although randomization rate ($\epsilon$) ...
- 1,283
2
votes
1
answer
147
views
How to prove the formula of eligibility traces operator in reinforcement learning?
I don't understand how the formula in the red circle is derived. The screenshot is taken from this paper
- 23
2
votes
0
answers
138
views
Why solely a one-step-lookahead in value/policy-iteration?
In value iteration and policy iteration we solely consider a one-step-lookahead where the lookahead is from the previous iteraiton and therefore need to sweep over all states and iterate this ...
- 53
2
votes
2
answers
200
views
How the proof of the contraction of variance for distributional Bellman operator follows
I am stuck at the proof of the contraction of variance for distributional Bellman operator from the paper, in which it is defined as
and the proof is stated as
In its second part, how is the ...
- 41
2
votes
1
answer
206
views
How are the Bellman optimality equations and minimax related?
Is the philosophy between Bellman equations and minimax the same?
Both the algorithms look at the full horizon and take into account potential gains (Bellman) and potential losses (minimax).
...
- 171
1
vote
1
answer
665
views
Connection between the Bellman equation for the action value function $q_\pi(s,a)$ and expressing $q_\pi(s,a) = q_\pi(s, a,v_\pi(s'))$
When deriving the Bellman equation for $q_\pi(s,a)$, we have
$q_\pi(s,a) = E_\pi[G_t | S_t = s, A_t = a] = E_\pi[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]$ (1)
This is what is confusing me, at this ...
- 382
1
vote
1
answer
85
views
How are these two equations for the optimal state-value function equivalent?
By substituting the optimal policy $\pi_{\star}$ into the Bellman equation, we get the Bellman equation for $v_{\pi_{\star}}(s)=v_{\star}(s)$:
$$ v_{\star}(s) = \sum\limits_a \pi_{\star}(a|s) \sum\...
- 754
1
vote
2
answers
195
views
Why $V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s,a),\forall s \in S$ in reinforcement learning?
In some RL notes, I encountered the following equation, which I am trying to prove:
$$
V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s, a),\forall s \in S
$$
Here is my attemption:
Firstly, I only need to ...
- 11
1
vote
1
answer
261
views
What is the difference between gradient decent in neural networks and temporal difference in reinforcement learning?
I am studying Q-learning in reinforcement learning. My question is about the Bellman equation.
In Q-learning, the Bellman equation is often introduced as follows.
\begin{align}
Q_{new}(s,a)
&= Q_{...
- 111
1
vote
1
answer
140
views
What would be the Bellman optimality equation for $q_∗(s, a)$ for an MDP with continuous states and actions?
I'm currently studying Reinforcement Learning and I'd like to know what would be the Bellman optimality equation for action values $q_∗(s, a)$ for a MDP with continuous states and actions, written out ...
- 145
1
vote
1
answer
248
views
How are these two versions of the Bellman optimality equation related?
I saw two versions of the optimality equation for $V_{*}(s)$ and $Q_{*}(s,a)$.
The first one is:
$$
V_{*}(s)=\max _{a} \sum_{s^{\prime}} P_{s s^{\prime}}^{a}\left(r(s, a)+\gamma V_{*}\left(s^{\prime}\...
- 49