# Questions tagged [bellman-equations]

For questions related to the Bellman equations in the context of reinforcement learning (and other artificial intelligence subfields).

68 questions
Filter by
Sorted by
Tagged with
1 vote
27 views

### Solving MDP as linear program: why minimize the sum of the states' values?

This is a follow-up question to the answer to How can we use linear programming to solve an MDP? Quick recap: the $max$ operators that appear in the Bellman optimality equations can be turned into a ...
39 views

### Direct solving of Bellman optimality equation [duplicate]

I'm reading the Sutton and Barto second edition (the 2020 revision). I'm on chapter 3.6 "Optimal Policies and Optimal Value Functions". It is suggested that the Bellman optimality equations ...
46 views

### Bellman equation for MRP

Bellman equation for MRP is: And it can be written as or the inverse matrix method, when the transition matrix is known. My question is: how the left $v$ equals to the right $v$ in the second ...
138 views

### Bellman equation and inverse matrix method

My problem: why the last formula do not contain information about time $t$? So if $s^{\prime}=s$, do we have $v_{\pi}(s) = v_{\pi}(s^{\prime})$? But this is not right I guess? If I am right, that they ...
1 vote
63 views

### When can we unnest the minimizations/recursions in an value function(bellman optimality equation)?

When reading the following paper(page 4): An Approximate Dynamic Programming Approach for Dual Stochastic Model Predictive Control I could see that they were able to unnest the minimization's in the ...
181 views

### How the proof of the contraction of variance for distributional Bellman operator follows

I am stuck at the proof of the contraction of variance for distributional Bellman operator from the paper, in which it is defined as and the proof is stated as In its second part, how is the ...
120 views

### Why solely a one-step-lookahead in value/policy-iteration?

In value iteration and policy iteration we solely consider a one-step-lookahead where the lookahead is from the previous iteraiton and therefore need to sweep over all states and iterate this ...
120 views

### Calculating state value using bellman equation question:

Consider a 4x4 grid world problem where the goal is to reach either the top left or bottom right corner. The agent can choose from four actions up,down,left,right which deterministically cause the ...
15 views

### Proof of Difference in Return Between Two Policies

I am attempting to understand why Lemma 6.1 holds in this paper on reinforcement learning. I have two questions. First, when defining the value function V(s), why is there a leading (1-γ) term? In the ...
65 views

### Could you explain these 2 steps of the derivation of the Bellman equation as a recursive equation in Sutton & Barto?

I am reading the Sutton & Barto (2018) RL textbook. On page 59, it derives the recursive property of the state-value function as below. Could you explain the steps of third and fourth equality? ...
214 views

### How do we get the optimal value-function?

In here it says that: (is it correct?) $$V^\pi = \sum_{a \in A}\pi(a|s)*Q^\pi(s,a)$$ And we have: $$V^*(s) = max_\pi V^\pi(s)$$ Also: $$V^*(s) = max_a Q^*(s, a)$$ Can someone demonstrate to me step ...
153 views

### How to prove that $V^\star$ is optimal if and only if it satisfies the Bellman equation?

The Question I'd like to prove that a function $V$ (like in reinforcement learning) is optimal iff it satisfies the bellman equation. A lot of places online reference this fact, but none prove it. ...
76 views

### Bellman optimality equation does not allow random policies?

I'm reading the Sutton & Barto's book "Reinforcement Learning: An Introduction" (2nd Edition). There is something I don't understand (p.63): ...
185 views

226 views

### How do we get the value of this state of an MDP, at time-step $h-2$, using dynamic programming?

I am trying to understand the problem below, represented as an MDP with four states (PU, PF, RU, and RF) and two actions (AS). Let's consider V(RF), the value of the state RF. At time-step $h$, V(RF) ...
When applying the bellman expectation equation: $$v(s)=\mathbb{E}\left[R_{t+1}+\gamma v\left(S_{t+1}\right) \mid S_{t}=s\right]$$ to the MRP below, states further away from the terminal state will ...