Skip to main content

Questions tagged [policy-evaluation]

For questions related to the various policy evaluation (PE) algorithms, which are numerical iterative algorithms that are used to find the value function associated with a given policy, which is often denoted as the "prediction problem". PE is also considered a dynamic programming method, which is regularly discussed in reinforcement learning textbooks.

Filter by
Sorted by
Tagged with
1 vote
0 answers
12 views

How to measure accuracy of learned value function of a fixed policy?

Let's say we've a given policy whose value function is to be evaluated. One way to get the value function can be using expected SARSA, as in this stack exchange answer. However, my MDP's state space ...
ModCon's user avatar
  • 111
1 vote
0 answers
23 views

How to modify the bellman operator for in-place iterative policy evaluation?

The iterative update rule for policy evaluation that is, approximating the value function for a given policy is: $$v^{k+1} = r_{\pi} + \gamma P_{\pi}v^{k}$$ This is the simultaneous update rule where ...
Atharva's user avatar
  • 11
1 vote
1 answer
54 views

Do I really need to do policy evaluation until convergence in policy iteration?

I am currently studying Sutton's book, and I learned that in policy iteration, policy evaluation is done until the value function converges, and then policy improvement is performed. However, I have ...
Jason's user avatar
  • 13
0 votes
1 answer
84 views

Why does my implementation of TD(0) not work?

I am trying to implement TD(0) among other RL Policy Evaluation techniques. I have also implemented the dynamic programming approach for a given model of the world and FV Monte Carlo and EV Monte ...
mavex857's user avatar
2 votes
1 answer
127 views

Is the existence and uniqueness of the state-value function for $\gamma < 1$ theoretical?

Consider the following statement from 4.1 Policy Evaluation of the first edition of Sutton and Barto's book. The existence and uniqueness of $V^{\pi}$ are guaranteed as long as either $\gamma < 1$...
hanugm's user avatar
  • 3,890
-1 votes
1 answer
53 views

Using states (features) and actions from a heuristic model to estimate the value function of a reinforcement learning agent [closed]

new to RL here. As far as i understood from RL courses, that there is two sides of reinforcement learning. Policy Evaluation, which is the task of knowing the value function for certain policy. and ...
Ramzy's user avatar
  • 3
1 vote
1 answer
86 views

Why is the update in-place faster than the out-of-place one in dynamic programming?

In Barto and Sutton's book, it's written that we have two types of updates in dynamic programming Update out-of-place Update in-place The update in-place is the faster one. Why is that the case? ...
VanasisB's user avatar
2 votes
1 answer
552 views

Can we use Q-learning update for policy evaluation (not control)?

For policy evaluation purposes, can we use the Q-learning algorithm even though, technically, it is meant for control? Maybe like this: Have the policy to be evaluated as the behaviour policy. Update ...
Dhruv Mullick's user avatar
2 votes
1 answer
586 views

Why do we need to go back to policy evaluation after policy improvement if the policy is not stable?

Above is the algorithm for Policy Iteration from Sutton's RL book. So, step 2 actually looks like value iteration, and then, at step 3 (policy improvement), if the policy isn't stable it goes back to ...
user8714896's user avatar
2 votes
1 answer
378 views

Is value iteration stopped after one update of each state?

In section 4.4 Value Iteration, the authors write One important special case is when policy evaluation is stopped after just one sweep (one update of each state). This algorithm is called value ...
Alex's user avatar
  • 23
4 votes
1 answer
613 views

Why is update rule of the value function different in policy evaluation and policy iteration?

In the textbook "Reinforcement Learning: An Introduction", by Richard Sutton and Andrew Barto, the pseudo code for Policy Evaluation is given as follows: The update equation for $V(s)$ comes from the ...
Jarvis1997's user avatar
9 votes
2 answers
5k views

What is the proof that policy evaluation converges to the optimal solution?

Although I know how the algorithm of iterative policy evaluation using dynamic programming works, I am having a hard time realizing how it actually converges. It appeals to intuition that, with each ...
SAGALPREET SINGH's user avatar
2 votes
1 answer
68 views

How can I implement policy evaluation when reward is tied to an action outcome?

I'm following Stanford reinforcement learning videos on youtube. One of the assignments asks to write code for policy evaluation for Gym's FrozenLake-v0 environment. In the course (and books I have ...
Argod's user avatar
  • 23
4 votes
3 answers
997 views

Why can the Bellman equation be turned into an update rule?

In chapter 4.1 of Sutton's book, the Bellman equation is turned into an update rule by simply changing the indices of it. How is it mathematically justified? I didn't quite get the initiation of why ...
Saeid Ghafouri's user avatar
1 vote
1 answer
306 views

Why isn't the implementation of my policy evaluation for a simple MDP converging?

I am trying to code out a policy evaluation algorithm to find the $V^\pi(s)$ for all states. The following diagram below shows the MDP. In this case i let p = q = 0.5. the rewards for each states ...
calveeen's user avatar
  • 1,281
4 votes
1 answer
695 views

How does policy evaluation work for continuous state space model-free approaches?

How does policy evaluation work for continuous state space model-free approaches? Theoretically, a model-based approach for the discrete state and action space can be computed via dynamic programming ...
calveeen's user avatar
  • 1,281
2 votes
0 answers
930 views

Difficulty understanding Monte Carlo policy evaluation (state-value) for gridworld

I've been trying to read Sutton & Barto book chapter 5.1, but I'm still a bit confused about the procedure of using Monte Carlo policy evaluation (p.92), and now I just cant proceed anymore coding ...
Late347's user avatar
  • 59
22 votes
2 answers
23k views

What is the difference between First-Visit Monte-Carlo and Every-Visit Monte-Carlo Policy Evaluation?

I came across these 2 algorithms, but I cannot understand the difference between these 2, both in terms of implementation as well as intuitionally. So, what difference does the second point in both ...
user avatar