6
votes
In Value Iteration, why can we initialize the value function arbitrarily?
Is this something to do with the Bellman optimality constraint itself?
That is part of it, and important for episodic problems without discounting. The Bellman equations link between time steps, ...
- 32.9k
5
votes
Accepted
Should the reward or the Q value be clipped for reinforcement learning
I'll start with the last question in your post:
I was also wondering if there are any theoretical proofs/explanations about reward/Q-value clipping and which one being better.
I highly doubt ...
- 10.4k
4
votes
Accepted
Why is my implementation of Q-learning not converging to the right values in the FrozenLake environment?
I was able to solve the problem.
The main issue for non-convergence was that I was not decaying the learning rate appropriately. I put a decay rate of $-0.00005$ on the learning rate ...
- 191
4
votes
Is the PyTorch official tutorial really about Q-learning?
TL;DR: It is Q learning. However Q learning is basically sample-based value iteration, so not surprising you see a similarity.
Q learning* and value iteration are very strongly related. When ...
- 32.9k
4
votes
Accepted
What is the difference between a reward and a value for a given state?
Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for ...
- 32.9k
4
votes
Accepted
Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?
You appear to comparing the value table update steps in policy iteration and value iteration, which are both derived from Bellman equations.
Policy iteration
In policy iteration, a policy lookup table ...
- 32.9k
4
votes
Accepted
In Value Iteration, why can we initialize the value function arbitrarily?
If the value function of a state $v(s)$ is relatively high, then you are absolutely correct in saying that a greedy policy may choose to visit $s$, since the high $v(s)$ makes it very promising. The ...
- 1,568
3
votes
Why do we need to go back to policy evaluation after policy improvement if the policy is not stable?
There is a difference between accurate value function estimates, and optimal value functions. An optimal value function is more specifically the value function of an optimal policy.
Value functions ...
- 32.9k
3
votes
Accepted
Value Iteration failing to converge to optimal value function in Sutton-Barto's Gambler problem
So, naturally, if you've observed something that contradicts the theoretical properties of Value Iteration, something's wrong, right?
Well, the code you've linked, as it is, is fine. It works as ...
- 436
3
votes
Accepted
Is value iteration stopped after one update of each state?
Where the author mentions the policy evaluation being stopped after one state, they are referring to the part of the algorithm that evaluates the policy -- the pseudocode you have listed is the ...
- 5,000
3
votes
What is the difference between a reward and a value for a given state?
What is the difference between a reward and a value for a given state?
Let us say that an agent took an action from state $A$ and reached state $B$ and got a score $R$. This instantaneous score the ...
3
votes
Would you categorize policy iteration as an actor-critic reinforcement learning approach?
Keeping this taxonomy intact for model-based Dynamic programming algorithms, I would argue that value iteration is a Actor only approach, and policy iteration is a Actor-Critic approach. However, not ...
- 32.9k
3
votes
What is generalized policy iteration?
In the standard policy iteration algorithm presented in Sutton and Barto's book, you alternate between a policy evaluation (PE) step and a policy improvement (PI) step (i.e. PE, PI, PE, PI, PE, PI, PE,...
- 41.1k
3
votes
What is the value of a state when there is a certain probability that agent will die after each step?
The value of a state depends on the policy that you use, so I'll make the assumption here that you're talking about value using the optimal policy.
According to the optimal policy, the agent would ...
- 66
3
votes
Why are policy iteration and value iteration studied as separate algorithms?
Policy iteration is made up of two steps. The first is a full policy evaluation, where a value function is calculated for the current policy. The second is policy improvement, where the policy is made ...
- 5,000
3
votes
Accepted
How is the fitted Q-iteration algorithm related to $Q^*(s, a)$, and how can we use function approximation with this algorithm?
1): The intuition is based on the concept of value iteration, which the authors mention but don't explain on page 504. The basic idea is this: imagine you knew the value of starting in state x and ...
- 9,342
3
votes
Why can't we apply value iteration when we do not know the reward and transition functions, and how does Q-learning solve this issue?
For normal value iteration, you need to have the model, i.e. the transition probability, denoted by $P(s' \mid s,a)$.
With Q-learning, you use the current reward and the already stored Q value:
The ...
- 375
3
votes
Accepted
A few questions regarding the difference between policy iteration and value iteration
$\pi(s)$ does not mean $q(s,a)$ here. $\pi(s)$ is a policy that represents probability distribution over action space for a specific state. $q(s,a)$ is a state-action pair value function that tells us ...
- 2,406
3
votes
Accepted
How do I apply the value iteration algorithm when there are two goal states?
What you could do is to trigger environment termination when rat either:
steps into the trap
picks both cheese pieces
The problem with such setup is that, when the rat picks a single piece, it ...
- 2,406
3
votes
Accepted
Why and how can the policy and value iteration methods converge to the OPTIMAL point?
These two algorithms converge to the optimal value function because
they are instances of the generalization policy iteration, so they iteratively perform one policy evaluation (PE) step followed by ...
- 41.1k
3
votes
Accepted
calculating the value of a state in an optimal policy analytically and iteratively
In the gridworld setting, you can replicate the lecture's results, by defining the reward function $R_t(s,a) = R(s)$, meaning that the reward function simply aggregates only on the current state and ...
- 929
3
votes
Accepted
What should the discount factor for the non-slippery version of the FrozenLake environment be?
After trying to understand what was happening by going through the algorithm on paper (below), I found all values were the same, so greedifying with respect to the value function often just had the ...
- 32.9k
3
votes
What are the similarities between Q-learning and Value Iteration?
Q learning is very similar to value iteration. They are based on the same principles. A key similarity is that both assume a greedy action choice on the bootstrap next state value.
The big difference ...
- 32.9k
2
votes
Accepted
Can policy iteration use only the immediate reward for updates?
Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?
Technically yes.
The value update step in Policy ...
- 32.9k
2
votes
Can I have different rewards for a single action based on which state it transitions to?
The reward function can be a function of the current state, current action, and next state: $R(s_t, a_t, s_{t+1})$. It's valid to use the Bellman operator in this setting because it's still a ...
2
votes
Accepted
Understanding the update rule for the policy in the policy iteration algorithm
A policy can be stochastic or deterministic. A deterministic policy is a function of the form $\pi_{\text{deterministic}}: S \rightarrow A$, that is, a function from the set of states to the set of ...
- 41.1k
2
votes
Accepted
What is the value of a state when there is a certain probability that agent will die after each step?
I will fill in some details in shaabhishek's answer for people who are interested.
With this in mind, what is the value of a square (1,1)?
First of all, the value function is dependent on a policy....
- 1,568
2
votes
Accepted
Unable to understand the second iteration update in value iteration algorithm for solving MDP
First thing to know is that, in this case, values for the gridworld in new iteration are completely calculated with respect to the old values from the previous iteration. Value of $0.78$ is got like ...
- 2,406
2
votes
Accepted
Why do value iteration and policy iteration obtain similar policies even though they have different value functions?
Both value iteration (VI) and policy iteration (PI) algorithms are guaranteed to converge to the optimal policy, so it is expected that you get similar policies from both algorithms (if they have ...
- 41.1k
2
votes
Why do I need an initial arbitrary policy to implement value iteration algorithm
It seems to me that you're thinking about the parameters a and b as being characteristic of the agent that's moving in the environment (therefore determining the final policy), but they are actually a ...
- 5,378
Only top scored, non community-wiki answers of a minimum length are eligible