Questions tagged [optimal-policy]
For questions related to the concept of "optimal policy" in reinforcement learning.
13
questions
0
votes
0
answers
20
views
What reward should be selected for transition states to make the agent reach the terminal state (destination) faster? negative, positive, or zero?
Consider the simple environment below, where the gray cells are the terminal states and the agent receives a reward of $-5$ for taking any action in these states. The nonterminal states are $S = \{1, ...
- 11
0
votes
1
answer
59
views
How is policy iteration capable of improving on a deterministic policy?
Given a policy $\pi$ and the improved version upon it using policy iteration $\pi'$ we have, for $\forall s \in S$, $v_{\pi'}(s)\geq v_{\pi}(s)$.
I think the way we choose $\pi'$ makes it ...
- 509
0
votes
0
answers
109
views
How do you determine the optimal policy?
I am following some Grid world examples to understand reinforcement learning. I have a deterministic grid (part of which I have reconstructed below). I am trying to understand how the optimal policy ...
- 1
0
votes
0
answers
54
views
Determine Gridworld values
I am learning Reinforcement learning for games following Gridworld examples. Apologies in advance if this is a basic question, very new to reinforcement learning.
I am slightly confused in scenarios ...
- 145
3
votes
1
answer
781
views
What is the difference between a greedy policy and an optimal policy?
I am struggling to understand what is the difference between an optimal policy and a greedy policy.
Let $F(r_{t+1},s_{t+1}| s_t,a_t)$ be the probability distribution accorting to which, given action $...
- 33
3
votes
1
answer
171
views
Can an optimal policy have a value function that has a smaller value for a state than a non-optimal policy?
I'm starting to learn about the Bellman Equation and a question came to my mind.
A policy $\pi$ is optimal if the value $v_\pi(s)$ is greater or equal than the value $v_{\pi'}(s)$ for all states $s \...
- 133
2
votes
1
answer
65
views
What does $v(S_{t+1})$ mean in the optimal state-action value function?
In Sutton & Barto's Reinforcement Learning: An Introduction page 63 the authors introduce the optimal state value function in the expression of the optimal action-value function as follows: $q_{*}(...
- 509
3
votes
1
answer
77
views
How is $v_*(s) = \max_{\pi} v_\pi(s)$ also applicable in the case of stochastic policies?
I am reading Sutton & Bartos's Book "Introduction to reinforcement learning". In this book, the defined the optimal value function as:
$$v_*(s) = \max_{\pi} v_\pi(s),$$ for all $s \in \...
- 33
1
vote
1
answer
51
views
Equivalence of the $Q(s,a)$ and $V(s)$ under optimality conditions?
Are the state-action values and the state value function equivalent for a given policy? I would assume so as the value function is defined as $V(s)=\sum_a \pi(a|s)Q_{\pi}(s,a)$. If we are operating a ...
5
votes
2
answers
696
views
Given two optimal policies, is an affine combination of them also optimal?
If there are two different optimal policies $\pi_1, \pi_2$ in a reinforcement learning task, will the linear combination (or affine combination) of the two policies $\alpha \pi_1 + \beta \pi_2, \alpha ...
- 53
5
votes
1
answer
642
views
What's the optimal policy in the rock-paper-scissors game?
A deterministic policy in the rock-paper-scissors game can be easily exploited by the opponent - by doing just the right sequence of moves to defeat the agent. More often than not, I've heard that a ...
- 1,021
1
vote
2
answers
362
views
Why is the optimal policy for an infinite horizon MDP deterministic?
Could someone please help me gain some intuition as to why the optimal policy for a Markov Decision Process in the infinite horizon case (agent acts forever) is deterministic?
- 1,021
4
votes
1
answer
295
views
An example of a unique value function which is associated with multiple optimal policies
In the 4th paragraph of
http://www.incompleteideas.net/book/ebook/node37.html
it is mentioned:
Whereas the optimal value functions for states and state-action pairs are unique for a given MDP, ...
- 143