# Questions tagged [bellman-equations]

For questions related to the Bellman equations in the context of reinforcement learning (and other artificial intelligence subfields).

66 questions
Filter by
Sorted by
Tagged with
16 views

### Bellman equation for MRP

Bellman equation for MRP is: And it can be written as or the inverse matrix method, when the transition matrix is known. My question is: how the left $v$ equals to the right $v$ in the second ...
117 views

### Bellman equation and inverse matrix method

My problem: why the last formula do not contain information about time $t$? So if $s^{\prime}=s$, do we have $v_{\pi}(s) = v_{\pi}(s^{\prime})$? But this is not right I guess? If I am right, that they ...
1 vote
57 views

### When can we unnest the minimizations/recursions in an value function(bellman optimality equation)?

When reading the following paper(page 4): An Approximate Dynamic Programming Approach for Dual Stochastic Model Predictive Control I could see that they were able to unnest the minimization's in the ...
63 views

### Why solely a one-step-lookahead in value/policy-iteration?

In value iteration and policy iteration we solely consider a one-step-lookahead where the lookahead is from the previous iteraiton and therefore need to sweep over all states and iterate this ...
75 views

### Calculating state value using bellman equation question:

Consider a 4x4 grid world problem where the goal is to reach either the top left or bottom right corner. The agent can choose from four actions up,down,left,right which deterministically cause the ...
14 views

### Proof of Difference in Return Between Two Policies

I am attempting to understand why Lemma 6.1 holds in this paper on reinforcement learning. I have two questions. First, when defining the value function V(s), why is there a leading (1-γ) term? In the ...
167 views

### How do we get the optimal value-function?

In here it says that: (is it correct?) $$V^\pi = \sum_{a \in A}\pi(a|s)*Q^\pi(s,a)$$ And we have: $$V^*(s) = max_\pi V^\pi(s)$$ Also: $$V^*(s) = max_a Q^*(s, a)$$ Can someone demonstrate to me step ...
56 views

### Could you explain these 2 steps of the derivation of the Bellman equation as a recursive equation in Sutton & Barto?

I am reading the Sutton & Barto (2018) RL textbook. On page 59, it derives the recursive property of the state-value function as below. Could you explain the steps of third and fourth equality? ...
55 views

### Bellman optimality equation does not allow random policies?

I'm reading the Sutton & Barto's book "Reinforcement Learning: An Introduction" (2nd Edition). There is something I don't understand (p.63): ...
133 views

### How to prove that $V^\star$ is optimal if and only if it satisfies the Bellman equation?

The Question I'd like to prove that a function $V$ (like in reinforcement learning) is optimal iff it satisfies the bellman equation. A lot of places online reference this fact, but none prove it. ...