Questions tagged [policies]
For questions related to policies (as defined in reinforcement learning or other AI sub-fields).
76
questions
1
vote
2
answers
506
views
Why does the policy $\pi$ affect the Q value?
From my understanding, the policy $\pi$ is basically how the agent acts (i.e. the actions it will take in each state).
However, I am confused about the Q value and how it is "affected" by a policy. ...
- 113
3
votes
0
answers
110
views
Representation of state space, action space and reward system for Reinforcement Learning problem
I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without ...
0
votes
0
answers
147
views
How to understand and visualize a trained RL agent's policy when the state space is high dimensional?
What are typical ways to understand and visualize a trained RL agent's policy when the state space is of high dimension (but not images)?
For example, suppose state and action are denoted by $s=(...
- 121
3
votes
3
answers
941
views
Is the policy really invariant under affine transformations of the reward function?
In the context of a Markov decision process, this paper says
it is well-known that the optimal policy is invariant to positive affine transformation of the reward function
On the other hand, ...
- 171
3
votes
0
answers
70
views
Can the importance sampling estimator have a non-stationary behaviour policy even if the target policy is stationary?
The inverse propensity score (IPS) estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained in the paper Doubly Robust Policy Evaluation and Optimization.
...
- 227
2
votes
0
answers
282
views
AlphaGo Zero: Does the policy head give a probability for every possible move?
If I understood correctly, the AlphaGo Zero network returns two values: a vector of logit probabilities p and a value v.
My question is: in this vector that it is outputted, do we have a probability ...
- 265
4
votes
1
answer
735
views
Why does having a fixed policy change a Markov Decision Process to a Markov Reward Process?
If a policy is fixed, it is said that a Markov Decision Process (MDP) becomes a Markov Reward Process (MRP).
Why is this so? Aren't the transitions and rewards still parameterized by the action and ...
- 43
4
votes
2
answers
437
views
Why is having low variance important in offline policy evaluation of reinforcement learning?
Intuitively, I understand that having an unbiased estimate of a policy is important because being biased just means that our estimate is distant from the truth value.
However, I don't understand ...
- 227
4
votes
1
answer
621
views
Can someone please help me validate my MDP?
Problem Statement :
I have a system with four states - S1 through S4 where S1 is the beginning state and S4 is the end/terminal state. The next state is always better than the previous state i.e if ...
- 83
1
vote
0
answers
161
views
Finding optimal Value function and Policy for an MDP
I am solving an RL MDP problem which is model based. I have an MDP which has four possible states S1-S4 and four different actions A1-A4, with S4 being terminal state and S1 is the beginning state. ...
- 83
1
vote
1
answer
98
views
Do I need to store the policy for RL?
I am creating a zero-sum game with RL and wondered if I need to store the policy, or if there are other RL methods that produce similar results (consistently beating the human player) without the need ...
- 613
2
votes
1
answer
61
views
If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy?
I wrote a simple implementation of Flappy Bird in Python, and now I'm trying to train an agent to play it at a reasonable skill level using TFLearn.
I feed the network an input vector of size 4:
...
- 53
8
votes
1
answer
7k
views
What is the difference between a stationary and a non-stationary policy?
In reinforcement learning, there are deterministic and non-deterministic (or stochastic) policies, but there are also stationary and non-stationary policies.
What is the difference between a ...
- 39.6k
8
votes
3
answers
16k
views
What is the difference between a stochastic and a deterministic policy?
In reinforcement learning, there are the concepts of stochastic (or probabilistic) and deterministic policies. What is the difference between them?
- 39.6k
2
votes
1
answer
101
views
Possible inconsistency in the Policy Improvement equation
I came across this formula in Sutton And Barto: RL an Intro (2nd Edition) equation number 4.7 (page number 78).
If $\pi$ and $\pi'$ are deterministic policies and $q_\pi(s, \pi'(s)) \geq v_\pi(s)$ ...
user9947
1
vote
0
answers
38
views
Measure grid-world environments difference for reinforcement learning
I'd like to measure the difference between 2 grid-worlds to determine the generalization capacity of my agent using tabular Q-learning.
Example (OpenAI Frozen Lake) :
SFFF
FHFH
FFFH
HFFG
and :
...
- 139
4
votes
1
answer
510
views
A few questions regarding the difference between policy iteration and value iteration [closed]
The question already has some answer. But I am still finding it quite unclear (also does $\pi(s)$ here mean $q(s,a)$ ?):
The few things I do not understand are:
Why the difference between 2 ...
user9947
3
votes
2
answers
406
views
What is the difference between a non-stationary policy and a state that stores time?
This question is related to What does "stationary" mean in the context of reinforcement learning?, but I have a more specific question to clarify the difference between a non-stationary ...
- 428
16
votes
3
answers
4k
views
Is the optimal policy always stochastic if the environment is also stochastic?
Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic?
Intuitively, if the environment is ...
- 39.6k
6
votes
1
answer
549
views
What is the relation between a policy which is the solution to a MDP and a policy like $\epsilon$-greedy?
In the context of reinforcement learning, a policy, $\pi$, is often defined as a function from the space of states, $\mathcal{S}$, to the space of actions, $\mathcal{A}$, that is, $\pi : \mathcal{S} \...
- 39.6k
5
votes
1
answer
136
views
How do I compute the variance of the return of an evaluation policy using two behaviour policies?
Suppose there is an evaluation policy called $\pi_{e}$ and there are two behavior policies $\pi_{b1}$ and $\pi_{b2}$. I know that it is possible to estimate the return of policy $\pi_{e}$ through ...
- 481
1
vote
0
answers
59
views
Where does the expectation term in the derivative of the soft-max policy come from?
At slide 17 of the David Silver's series, the soft-max policy is defined as follows
$$
\pi_\theta(s, a) \propto e^{\phi(s, a)^T \theta}
$$
that is, the probability of an action $a$ (in state $s$) is ...
- 253
1
vote
1
answer
153
views
Why does the value of state change depending on the policy used to get to that state?
From what I understand, the value function estimates how 'good' it is for an agent to be in a state, and a policy is a mapping of actions to state.
If I have understood these concepts correctly, why ...
- 113
5
votes
2
answers
618
views
Why is the derivative of this objective function 0 if the policy is deterministic?
In the Berkeley RL class CS294-112 Fa18 9/5/18, they mention the following gradient would be 0 if the policy is deterministic.
$$
\nabla_{\theta} J(\theta)=E_{\tau \sim \pi_{\theta}(\tau)}\left[\left(\...
- 153
15
votes
4
answers
7k
views
What does "stationary" mean in the context of reinforcement learning?
I think I've seen the expressions "stationary data", "stationary dynamics" and "stationary policy", among others, in the context of reinforcement learning. What does it mean? I think stationary policy ...
- 428
4
votes
1
answer
329
views
An example of a unique value function which is associated with multiple optimal policies
In the 4th paragraph of
http://www.incompleteideas.net/book/ebook/node37.html
it is mentioned:
Whereas the optimal value functions for states and state-action pairs are unique for a given MDP, ...
- 143