Skip to main content
Share Your Experience: Take the 2024 Developer Survey
6 votes

In Value Iteration, why can we initialize the value function arbitrarily?

Is this something to do with the Bellman optimality constraint itself? That is part of it, and important for episodic problems without discounting. The Bellman equations link between time steps, ...
Neil Slater's user avatar
  • 32.5k
5 votes
Accepted

Should the reward or the Q value be clipped for reinforcement learning

I'll start with the last question in your post: I was also wondering if there are any theoretical proofs/explanations about reward/Q-value clipping and which one being better. I highly doubt ...
Dennis Soemers's user avatar
  • 10.3k
4 votes
Accepted

Why is my implementation of Q-learning not converging to the right values in the FrozenLake environment?

I was able to solve the problem. The main issue for non-convergence was that I was not decaying the learning rate appropriately. I put a decay rate of $-0.00005$ on the learning rate ...
abkds's user avatar
  • 191
4 votes

Is the PyTorch official tutorial really about Q-learning?

TL;DR: It is Q learning. However Q learning is basically sample-based value iteration, so not surprising you see a similarity. Q learning* and value iteration are very strongly related. When ...
Neil Slater's user avatar
  • 32.5k
4 votes
Accepted

What is the difference between a reward and a value for a given state?

Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for ...
Neil Slater's user avatar
  • 32.5k
4 votes
Accepted

Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?

You appear to comparing the value table update steps in policy iteration and value iteration, which are both derived from Bellman equations. Policy iteration In policy iteration, a policy lookup table ...
Neil Slater's user avatar
  • 32.5k
4 votes
Accepted

In Value Iteration, why can we initialize the value function arbitrarily?

If the value function of a state $v(s)$ is relatively high, then you are absolutely correct in saying that a greedy policy may choose to visit $s$, since the high $v(s)$ makes it very promising. The ...
DeepQZero's user avatar
  • 1,399
3 votes

Why do we need to go back to policy evaluation after policy improvement if the policy is not stable?

There is a difference between accurate value function estimates, and optimal value functions. An optimal value function is more specifically the value function of an optimal policy. Value functions ...
Neil Slater's user avatar
  • 32.5k
3 votes
Accepted

Value Iteration failing to converge to optimal value function in Sutton-Barto's Gambler problem

So, naturally, if you've observed something that contradicts the theoretical properties of Value Iteration, something's wrong, right? Well, the code you've linked, as it is, is fine. It works as ...
Asher's user avatar
  • 436
3 votes
Accepted

Is value iteration stopped after one update of each state?

Where the author mentions the policy evaluation being stopped after one state, they are referring to the part of the algorithm that evaluates the policy -- the pseudocode you have listed is the ...
David's user avatar
  • 4,920
3 votes

What is the difference between a reward and a value for a given state?

What is the difference between a reward and a value for a given state? Let us say that an agent took an action from state $A$ and reached state $B$ and got a score $R$. This instantaneous score the ...
Sooryakiran Pallikulathil's user avatar
3 votes

Would you categorize policy iteration as an actor-critic reinforcement learning approach?

Keeping this taxonomy intact for model-based Dynamic programming algorithms, I would argue that value iteration is a Actor only approach, and policy iteration is a Actor-Critic approach. However, not ...
Neil Slater's user avatar
  • 32.5k
3 votes

What is generalized policy iteration?

In the standard policy iteration algorithm presented in Sutton and Barto's book, you alternate between a policy evaluation (PE) step and a policy improvement (PI) step (i.e. PE, PI, PE, PI, PE, PI, PE,...
nbro's user avatar
  • 40.8k
3 votes

What is the value of a state when there is a certain probability that agent will die after each step?

The value of a state depends on the policy that you use, so I'll make the assumption here that you're talking about value using the optimal policy. According to the optimal policy, the agent would ...
shaabhishek's user avatar
3 votes

Why are policy iteration and value iteration studied as separate algorithms?

Policy iteration is made up of two steps. The first is a full policy evaluation, where a value function is calculated for the current policy. The second is policy improvement, where the policy is made ...
David's user avatar
  • 4,920
3 votes
Accepted

How is the fitted Q-iteration algorithm related to $Q^*(s, a)$, and how can we use function approximation with this algorithm?

1): The intuition is based on the concept of value iteration, which the authors mention but don't explain on page 504. The basic idea is this: imagine you knew the value of starting in state x and ...
John Doucette's user avatar
3 votes

Why can't we apply value iteration when we do not know the reward and transition functions, and how does Q-learning solve this issue?

For normal value iteration, you need to have the model, i.e. the transition probability, denoted by $P(s' \mid s,a)$. With Q-learning, you use the current reward and the already stored Q value: The ...
agold's user avatar
  • 375
3 votes
Accepted

A few questions regarding the difference between policy iteration and value iteration

$\pi(s)$ does not mean $q(s,a)$ here. $\pi(s)$ is a policy that represents probability distribution over action space for a specific state. $q(s,a)$ is a state-action pair value function that tells us ...
Brale's user avatar
  • 2,406
3 votes
Accepted

How do I apply the value iteration algorithm when there are two goal states?

What you could do is to trigger environment termination when rat either: steps into the trap picks both cheese pieces The problem with such setup is that, when the rat picks a single piece, it ...
Brale's user avatar
  • 2,406
3 votes
Accepted

Why and how can the policy and value iteration methods converge to the OPTIMAL point?

These two algorithms converge to the optimal value function because they are instances of the generalization policy iteration, so they iteratively perform one policy evaluation (PE) step followed by ...
nbro's user avatar
  • 40.8k
3 votes
Accepted

calculating the value of a state in an optimal policy analytically and iteratively

In the gridworld setting, you can replicate the lecture's results, by defining the reward function $R_t(s,a) = R(s)$, meaning that the reward function simply aggregates only on the current state and ...
ddaedalus's user avatar
  • 919
3 votes
Accepted

What should the discount factor for the non-slippery version of the FrozenLake environment be?

After trying to understand what was happening by going through the algorithm on paper (below), I found all values were the same, so greedifying with respect to the value function often just had the ...
Neil Slater's user avatar
  • 32.5k
3 votes

What are the similarities between Q-learning and Value Iteration?

Q learning is very similar to value iteration. They are based on the same principles. A key similarity is that both assume a greedy action choice on the bootstrap next state value. The big difference ...
Neil Slater's user avatar
  • 32.5k
2 votes
Accepted

Can policy iteration use only the immediate reward for updates?

Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function? Technically yes. The value update step in Policy ...
Neil Slater's user avatar
  • 32.5k
2 votes

Can I have different rewards for a single action based on which state it transitions to?

The reward function can be a function of the current state, current action, and next state: $R(s_t, a_t, s_{t+1})$. It's valid to use the Bellman operator in this setting because it's still a ...
Mehran Shakerinava's user avatar
2 votes
Accepted

Understanding the update rule for the policy in the policy iteration algorithm

A policy can be stochastic or deterministic. A deterministic policy is a function of the form $\pi_{\text{deterministic}}: S \rightarrow A$, that is, a function from the set of states to the set of ...
nbro's user avatar
  • 40.8k
2 votes
Accepted

What is the value of a state when there is a certain probability that agent will die after each step?

I will fill in some details in shaabhishek's answer for people who are interested. With this in mind, what is the value of a square (1,1)? First of all, the value function is dependent on a policy....
DeepQZero's user avatar
  • 1,399
2 votes
Accepted

Unable to understand the second iteration update in value iteration algorithm for solving MDP

First thing to know is that, in this case, values for the gridworld in new iteration are completely calculated with respect to the old values from the previous iteration. Value of $0.78$ is got like ...
Brale's user avatar
  • 2,406
2 votes
Accepted

Why do value iteration and policy iteration obtain similar policies even though they have different value functions?

Both value iteration (VI) and policy iteration (PI) algorithms are guaranteed to converge to the optimal policy, so it is expected that you get similar policies from both algorithms (if they have ...
nbro's user avatar
  • 40.8k
2 votes

Why do I need an initial arbitrary policy to implement value iteration algorithm

It seems to me that you're thinking about the parameters a and b as being characteristic of the agent that's moving in the environment (therefore determining the final policy), but they are actually a ...
Edoardo Guerriero's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible