Questions tagged [value-iteration]

For questions related to the value iteration algorithm, which is a dynamic programming (DP) algorithm used to solve an MDP, that is, it is used to find a policy given the transition and reward functions of the MDP. Value iteration is related to another DP algorithm called policy iteration.

Filter by
Sorted by
Tagged with
2
votes
1answer
58 views

Is the PyTorch official tutorial really about Q-learning?

I read Q-learning algorithm and also I know value iteration (when you update action values). I think the PyTorch example is value iteration rather than Q-learning. Here is the link: https://pytorch....
3
votes
1answer
40 views

Would you categorize policy iteration as an actor-critic reinforcement learning approach?

One way of understanding the difference between value function approaches, policy approaches and actor-critic approaches in reinforcement learning is the following: A critic explicitly models a value ...
1
vote
1answer
26 views

What is generalized policy iteration?

I am reading Sutton and Barto's material now. I know value iteration, which is an iterative algorithm taking the maximum value of adjacent states, and policy iteration. But what is generalized policy ...
2
votes
1answer
42 views

Why do value iteration and policy iteration obtain similar policies even though they have different value functions?

I am trying to implement value and policy iteration algorithms. My value function from policy iteration looks vastly different from the values from value iteration, but the policy obtained from both ...
2
votes
1answer
50 views

Why do I need an initial arbitrary policy to implement value iteration algorithm

I've been recently given an assignment based on Reinforcement Learning and I'm supposed to implement the value iteration algorithm in a grid environment. The assignment: My doubt is why do I even ...
0
votes
1answer
13 views

Unable to understand V* at infinite time horizon using Bellman equation for solving MDP

I've been following the Berkeley cs188's assignment (I'm not taking the course). Currently, they don't show the solution in the gradescope unless I get it correct. My reasoning was $V^*(a)$ = 10 ...
0
votes
0answers
62 views

What is the difference between value iteration and policy iteration? [duplicate]

In reinforcement learning, what is the difference between policy iteration and value iteration? As much as I understand, in value iteration, you use the Bellman equation to solve for the optimal ...
2
votes
1answer
247 views

Q Learning for FrozenLake environment not converging to V* values from Value Iteration

I am trying to learn tabular Q learning, value iteration using the classical algorithms (no neural networks) by using a table of states and actions. I was trying it out on FrozenLake environment in ...
1
vote
0answers
66 views

Is Value Iteration better than Policy Iteration for first few iterations?

In Policy Iteration (PI), the action generated by the policy, whether it's optimal or not w.r.t the current value function $v(s)$. Whereas, in Value Iteration, the action is greedily generated w.r.t ...
2
votes
1answer
35 views

Can policy iteration use only the immediate reward for updates?

Is it still a policy iteration algorithm if the policy is updated optimizing a function of the immediate reward instead of the value function?
3
votes
1answer
48 views

Can I have different rewards for a single action based on which state it transitions to?

I am working on an MDP where there are four states and ten actions. I am supposed to derive the optimal policy to reach the desired state. At any state, a particular action can take you to any of the ...
3
votes
1answer
256 views

Understanding the update rule for the policy in the policy iteration algorithm

Consider the grid world problem in RL. Formally, policy in RL is defined as $\pi(a|s)$. If we are solving grid world by policy iteration then the following pseudocode is used: My question is related ...
1
vote
1answer
192 views

Unable to understand the second iteration update in value iteration algorithm for solving MDP

I am trying to understand the value iteration method for Markov Decision Process(MDP) and I was referring ot UC Berkeley's slides titled Markov Decision Processes and Exact Solution Methods On slide ...
4
votes
2answers
243 views

A few questions regarding the difference between policy iteration and value iteration

The question already has some answer. But I am still finding it quite unclear (also does $\pi(s)$ here mean $q(s,a)$ ?): The few things I do not understand are: Why the difference between 2 ...
1
vote
1answer
118 views

How do I apply the value iteration algorithm when there are two goal states?

I am working through the famous RL textbook by Sutton & Barto. Currently, I am on the value iteration chapter. To gain better understanding, I coded up a small example, inspired by this article. ...
2
votes
1answer
2k views

What is the time complexity of the value iteration algorithm?

Recently, I have come across the information (lecture 8 and 9 about MDPs of this UC Berkeley AI course) that the time complexity for each iteration of the value iteration algorithm is $\mathcal{O}(|S|^...
4
votes
1answer
1k views

Should the reward or the Q value be clipped for reinforcement learning

When extending reinforcement learning to the continuous states, continuous action case, we must use function approximators (linear or non-linear) to approximate the Q-value. It is well known that non-...
3
votes
1answer
81 views

In reinforcement learning, Is the optimal value (V*) corresponding to performing the best action in a given state?

it seems I am a little confused about the optimal value (V*) and optimal action-value (Q*) in reinforcement learning and just want some clarity because some blogs I read on Medium and GitHub are ...
5
votes
1answer
122 views

Reinforcement Learning (Fitted Q): Qn on Concept & Implementation

I hope to get some clarifications on Fitted Q-Learning ('FQL'). My Research So Far I've read Sutton's book (specifically, chp 6 to 10), Ernst et al and this paper. I know that ...