Questions tagged [bellman-equations]
For questions related to the Bellman equations in the context of reinforcement learning (and other artificial intelligence subfields).
66
questions
0
votes
0
answers
16
views
Bellman equation for MRP
Bellman equation for MRP is:
And it can be written as
or the inverse matrix method, when the transition matrix is known. My question is: how the left $v$ equals to the right $v$ in the second ...
- 21
2
votes
1
answer
117
views
Bellman equation and inverse matrix method
My problem: why the last formula do not contain information about time $t$?
So if $s^{\prime}=s$, do we have $v_{\pi}(s) = v_{\pi}(s^{\prime})$? But this is not right I guess?
If I am right, that they ...
- 21
1
vote
0
answers
57
views
When can we unnest the minimizations/recursions in an value function(bellman optimality equation)?
When reading the following paper(page 4): An Approximate Dynamic Programming Approach
for Dual Stochastic Model Predictive Control
I could see that they were able to unnest the minimization's in the ...
- 11
2
votes
0
answers
63
views
Why solely a one-step-lookahead in value/policy-iteration?
In value iteration and policy iteration we solely consider a one-step-lookahead where the lookahead is from the previous iteraiton and therefore need to sweep over all states and iterate this ...
- 53
0
votes
1
answer
75
views
Calculating state value using bellman equation question:
Consider a 4x4 grid world problem where the goal is to reach either the top left or bottom right corner. The agent can choose from four actions up,down,left,right which deterministically cause the ...
- 23
0
votes
0
answers
14
views
Proof of Difference in Return Between Two Policies
I am attempting to understand why Lemma 6.1 holds in this paper on reinforcement learning. I have two questions. First, when defining the value function V(s), why is there a leading (1-γ) term? In the ...
3
votes
2
answers
167
views
How do we get the optimal value-function?
In here it says that: (is it correct?)
$$V^\pi = \sum_{a \in A}\pi(a|s)*Q^\pi(s,a)$$
And we have:
$$ V^*(s) = max_\pi V^\pi(s)$$
Also:
$$ V^*(s) = max_a Q^*(s, a) $$
Can someone demonstrate to me step ...
- 206
4
votes
1
answer
56
views
Could you explain these 2 steps of the derivation of the Bellman equation as a recursive equation in Sutton & Barto?
I am reading the Sutton & Barto (2018) RL textbook.
On page 59, it derives the recursive property of the state-value function as below.
Could you explain the steps of third and fourth equality?
...
- 195
0
votes
0
answers
55
views
Bellman optimality equation does not allow random policies?
I'm reading the Sutton & Barto's book "Reinforcement Learning: An Introduction" (2nd Edition). There is something I don't understand (p.63):
...
0
votes
1
answer
133
views
How to prove that $V^\star$ is optimal if and only if it satisfies the Bellman equation?
The Question
I'd like to prove that a function $V$ (like in reinforcement learning) is optimal iff it satisfies the bellman equation. A lot of places online reference this fact, but none prove it. ...
- 79
2
votes
2
answers
172
views
How the proof of the contraction of variance for distributional Bellman operator follows
I am stuck at the proof of the contraction of variance for distributional Bellman operator from the paper, in which it is defined as
and the proof is stated as
In its second part, how is the ...
- 41
3
votes
1
answer
170
views
Why does A2C use the actual returns from an episode in calculating the advantage?
Why does A2C use the actual returns from an episode in calculating the advantage instead of using a bellman equation style estimate of the value?
Basically, why this:
$A(s,a) = \sum_t\gamma^tr_t - V(s)...
- 125
1
vote
1
answer
78
views
How are these two equations for the optimal state-value function equivalent?
By substituting the optimal policy $\pi_{\star}$ into the Bellman equation, we get the Bellman equation for $v_{\pi_{\star}}(s)=v_{\star}(s)$:
$$ v_{\star}(s) = \sum\limits_a \pi_{\star}(a|s) \sum\...
- 493
1
vote
2
answers
138
views
Why $V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s,a),\forall s \in S$ in reinforcement learning?
In some RL notes, I encountered the following equation, which I am trying to prove:
$$
V^{\pi^*}(s) = \max_{a \in A}Q^{\pi^*}(s, a),\forall s \in S
$$
Here is my attemption:
Firstly, I only need to ...
- 11
2
votes
1
answer
140
views
What is the difference between these two versions of the Bellman equation?
The first version is the one I am most familiar with:
$$V_\pi(s) = \sum_{a}^{}\pi(a|s) \sum_{s'}^{}T(s, a, s')[R(s, a, s') + \gamma V_\pi(s')]$$
where $T(s, a, s')$ represents the probability of ...
- 133
2
votes
1
answer
791
views
Are my proofs that the Bellman operators are contractions correct?
Introduction
I'm studying Reinforcement Learning, and in order to increase my understanding I've been challenging myself by trying to write proofs that show that the right hand side of the Bellman ...
- 53
1
vote
0
answers
26
views
Proof that the Policy Iteration Converges?
Let $\mathcal{X}=:\{x_1, x_2, x_3,...,x_n\}$ be the state space. Let $\mathcal{U}:=\{u_1, u_2, u_3,...,u_m\}$ be the set of actions. Let $A^{u_1}, A^{u_2}, A^{u_3},...,A^{u_m}$ be the state transition ...
1
vote
1
answer
131
views
What would be the Bellman optimality equation for $q_∗(s, a)$ for an MDP with continuous states and actions?
I'm currently studying Reinforcement Learning and I'd like to know what would be the Bellman optimality equation for action values $q_∗(s, a)$ for a MDP with continuous states and actions, written out ...
- 145
0
votes
1
answer
156
views
Is my derivation of the Bellman equation for $q_{\pi}$ in terms of $p(s'|s,a)$ and $r(s,a)$ correct?
I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:
Rewrite the Bellman equation for the function $q_{\pi}$ in terms of the three argument ...
- 145
1
vote
0
answers
15
views
Value iteration algorithm converge to max reward with discount of 1? [duplicate]
I'm running the value iteration algorithm of a gridworld of 4*3, with two terminal nodes with -50 reward and one with +20 reward like so:
...
- 165
3
votes
1
answer
70
views
Can we also estimate $V_{\pi}$ with SARSA?
For SARSA, I know we can estimate the action value $Q(s,a)$, and the relationship between $V(s)$ and $Q(s,a)$ is $V_{\pi}(s) = \sum_{a \in \mathcal{A}} \pi(a|s)Q_{\pi}
(s,a)$.
So my question is, can ...
- 31
-1
votes
1
answer
777
views
What is so special about the Bellman Optimality Principle?
In the context of Decision Making and Game Theory, "Bellman's Equations and Bellman's Conditions of Optimality" are said to be some of the most important mathematical principles in this ...
- 269
2
votes
1
answer
284
views
Determining to terminate at a reward or not
I am practicing the Bellman equation on Grid world examples and in this scenario, there are numbered grid squares where the agent can choose to terminate and collect the reward equal to the amount ...
- 145
2
votes
2
answers
913
views
What is the Bellman equation for V(s) in the case of a deterministic environment?
I am currently trying to practice reinforcement learning for an agent on a grid. The grid is deterministic. Since the grid is deterministic, to calculate the value for each grid square from the reward ...
- 145
0
votes
0
answers
175
views
Determine Gridworld values
I am learning Reinforcement learning for games following Gridworld examples. Apologies in advance if this is a basic question, very new to reinforcement learning.
I am slightly confused in scenarios ...
- 145
3
votes
1
answer
2k
views
What is the difference between a greedy policy and an optimal policy?
I am struggling to understand what is the difference between an optimal policy and a greedy policy.
Let $F(r_{t+1},s_{t+1}| s_t,a_t)$ be the probability distribution accorting to which, given action $...
- 33
2
votes
1
answer
392
views
calculating the value of a state in an optimal policy analytically and iteratively
I am watching the lecture by Abbeel on MDPs and Reinforcement Learning. The setup of the problem is the classic gridworld with optimal policy (and corresponding values of states) pictured below.
The ...
- 175
2
votes
1
answer
284
views
Where are the parentheses in the Bellman update rule?
I'm not having a lot of intuition about the equation. I have this Bellman update rule:
$$v_{\pi}(s) =\sum_a \pi(a|s)\sum_{s',r} p(s',r|s,a)[r+ \gamma v_{k}(s')]$$
But where are the parenthesis? Is the ...
- 206
1
vote
1
answer
56
views
Would it be possible to enforce the same $s_{t + 1}$ between the model's estimate and the target function's Q-value?
Say I have a game of blackjack, and I am trying to teach a single forward-pass neural network to approximate the Q value of the current state and action.
There are 3 inputs: The current card in hand, ...
- 37
3
votes
1
answer
339
views
Can an optimal policy have a value function that has a smaller value for a state than a non-optimal policy?
I'm starting to learn about the Bellman Equation and a question came to my mind.
A policy $\pi$ is optimal if the value $v_\pi(s)$ is greater or equal than the value $v_{\pi'}(s)$ for all states $s \...
- 133
1
vote
1
answer
225
views
What is the difference between gradient decent in neural networks and temporal difference in reinforcement learning?
I am studying Q-learning in reinforcement learning. My question is about the Bellman equation.
In Q-learning, the Bellman equation is often introduced as follows.
\begin{align}
Q_{new}(s,a)
&= Q_{...
- 111
1
vote
1
answer
186
views
How are these two versions of the Bellman optimality equation related?
I saw two versions of the optimality equation for $V_{*}(s)$ and $Q_{*}(s,a)$.
The first one is:
$$
V_{*}(s)=\max _{a} \sum_{s^{\prime}} P_{s s^{\prime}}^{a}\left(r(s, a)+\gamma V_{*}\left(s^{\prime}\...
- 49
3
votes
1
answer
303
views
Why must the value of a state under an optimal policy equal the expected return for the best action from that state?
The Sutton and Barto reinforcement learning textbook states that
the value of a state under an optimal policy must equal the expected return for the best action from that state.
That is,
$$v_*(s) = \...
3
votes
2
answers
963
views
How can we find the value function by solving a system of linear equations?
I am following the book "Reinforcement Learning: An Introduction" by Richard Sutton and Andrew Barto, and they give an example of a problem for which the value function can be computed ...
- 143
4
votes
1
answer
2k
views
What do the terms 'Bellman backup' and 'Bellman error' mean?
Some RL literature use terms such as: 'Bellman backup' and 'Bellman error'. What do these terms refer to?
- 359
4
votes
1
answer
142
views
How to prove the second form of Bellman's equation?
I'd like to prove this "second form" of Bellman's equation: $v(s) = \mathbb{E}[R_{t + 1} + \gamma v(S_{t+1}) \mid S_{t} = s]$ starting from Bellman's equation: $v(s) = \mathbb{E}[G_{t} \mid ...
- 573
0
votes
1
answer
221
views
How do we get the value of this state of an MDP, at time-step $h-2$, using dynamic programming?
I am trying to understand the problem below, represented as an MDP with four states (PU, PF, RU, and RF) and two actions (AS).
Let's consider V(RF), the value of the state RF. At time-step $h$, V(RF) ...
- 61
0
votes
1
answer
200
views
Converging to a wrong optimal policy if the agent is given more choices
I am a bit new to Reinforcement learning. So, I am extremely sorry if I am asking something obvious. I have written a small piece of code to find the optimal policy for a 5x5 grid problem.
Scenario 1....
- 3
-2
votes
1
answer
774
views
Why the optimal Bellman operator of a Q-function can be approximated by a single point
I am currently studying reinforcement learning, especially DQN.
In DQN, learning proceeds in such a way as to minimize the norm (least-squares, Huber, etc.) of the optimal Bellman equation and the ...
- 97
1
vote
1
answer
563
views
Bellman Expectation Equation leading to results where value iteration would not converge to the optimal policy
When applying the bellman expectation equation:
$$v(s)=\mathbb{E}\left[R_{t+1}+\gamma v\left(S_{t+1}\right) \mid S_{t}=s\right]$$
to the MRP below, states further away from the terminal state will ...
- 19
2
votes
1
answer
2k
views
How to avoid being stuck local optima in q-learning and q-network
When using the Bellman equation to update q-table or train q-network to fit greedy max values, the q-values very often get to the local optima and get stuck although randomization rate ($\epsilon$) ...
- 1,283
5
votes
1
answer
950
views
How would I compute the optimal state-action value for a certain state and action?
I am currently trying to learn reinforcement learning and I started with the basic gridworld application. I tried Q-learning with the following parameters:
Learning rate = 0.1
Discount factor = 0.95
...
- 215
3
votes
1
answer
327
views
How can we find the value function by solving a system of linear equations without knowing the policy?
An MDP is a Markov Reward Process with decisions, it’s an environment in which all states are Markov. This is what we want to solve. An MDP is a tuple $(S, A, P, R, \gamma)$, where $S$ is our state ...
- 45
2
votes
1
answer
146
views
How to prove the formula of eligibility traces operator in reinforcement learning?
I don't understand how the formula in the red circle is derived. The screenshot is taken from this paper
- 23
2
votes
1
answer
664
views
How to derive matrix form of the Bellman operators?
Reading the Retrace paper (Safe and efficient off-policy reinforcement learning) I saw they often use a matrix form of the Bellman operators, for example as in the picture below. How do we derive ...
- 233
6
votes
2
answers
2k
views
What is the Bellman Equation actually telling?
What does the Bellman equation actually say? And are there many flavours of that?
I get a little confused when I look for the Bellman equation, because I feel like people are telling slightly ...
- 69
0
votes
1
answer
72
views
Are the state-action values and the state value function equivalent for a given policy?
Are the state-action values and the state value function equivalent for a given policy? I would assume so as the value function is defined as $V(s)=\sum_a \pi(a|s)Q_{\pi}(s,a)$. If we are operating a ...
8
votes
1
answer
1k
views
How is the DQN loss derived from (or theoretically motivated by) the Bellman equation, and how is it related to the Q-learning update?
I'm doing a project on Reinforcement Learning. I programmed an agent that uses DDQN. There are a lot of tutorials on that, so the code implementation was not that hard.
However, I have problems ...
- 183
4
votes
1
answer
394
views
How are afterstate value functions mathematically defined?
In this answer, afterstate value functions are mentioned, and that temporal-difference (TD) and Monte Carlo (MC) methods can also use these value functions. Mathematically, how are these value ...
- 39.7k
4
votes
1
answer
145
views
Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?
I was looking at the Bellman equation, and I noticed a difference between the equations used in policy evaluation and value iteration.
In policy evaluation, there was the presence of $\pi(a \mid s)$, ...
- 125