We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [policy]

For questions related to policies (as defined in reinforcement learning or other AI sub-fields).

Filter by
Sorted by
Tagged with
10
votes
4answers
1k views

Is the optimal policy always stochastic if the environment is also stochastic?

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? Intuitively, if the environment is ...
7
votes
2answers
482 views

What does “stationary” mean in the context of reinforcement learning?

I think I've seen the expressions "stationary data", "stationary dynamics" and "stationary policy", among others, in the context of reinforcement learning. What does it mean? I think stationary policy ...
5
votes
1answer
98 views

How do I compute the variance of the return of an evaluation policy using two behaviour policies?

Suppose there is an evaluation policy called $\pi_{e}$ and there are two behavior policies $\pi_{b1}$ and $\pi_{b2}$. I know that it is possible to estimate the return of policy $\pi_{e}$ through ...
4
votes
2answers
58 views

Why is having low variance important in offline policy evaluation of reinforcement learning?

Intuitively, I understand that having an unbiased estimate of a policy is important because being biased just means that our estimate is distant from the truth value. However, I don't understand ...
3
votes
2answers
102 views

What is the difference between a non-stationary policy and a state that stores time?

This question is related to What does "stationary" mean in the context of reinforcement learning?, but I have a more specific question to clarify the difference between a non-stationary ...
3
votes
1answer
147 views

What is Policy Iteration in RL?

Consider the Gridworld problem in RL. Formally, policy in RL is defined as $\pi(a|s)$. If we are solving Gridworld by Policy Iteration then the following pseudocode is used: Now the question is in ...
3
votes
2answers
141 views

A few questions regarding the difference between policy iteration and value iteration

The question already has some answer. But I am still finding it quite unclear (also does $\pi(s)$ here mean $q(s,a)$ ?): The few things I do not understand are: Why the difference between 2 ...
3
votes
1answer
31 views

Why does having a fixed policy change a Markov Decision Process (MDP) to a Markov Reward Process (MRP)?

If a policy is fixed, it is said that an MDP becomes an MRP. Why is this so? Aren't the transitions and rewards still parameterized by the action and current state? In other words, aren't the ...
3
votes
1answer
157 views

An example of a unique value function which is associated with multiple optimal policies

In the 4th paragraph of http://www.incompleteideas.net/book/ebook/node37.html it is mentioned: Whereas the optimal value functions for states and state-action pairs are unique for a given MDP, ...
2
votes
1answer
55 views

Can someone please help me validate my MDP?

Problem Statement : I have a system with four states - S1 through S4 where S1 is the beginning state and S4 is the end/terminal state. The next state is always better than the previous state i.e if ...
2
votes
1answer
41 views

If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy?

I wrote a simple implementation of Flappy Bird in Python, and now I'm trying to train an agent to play it at a reasonable skill level using TFLearn. I feed the network an input vector of size 4: ...
2
votes
1answer
62 views

What is the relation between a policy which is the solution to a MDP and a policy like $\epsilon$-greedy?

In the context of reinforcement learning, a policy, $\pi$, is often defined as a function from the space of states, $\mathcal{S}$, to the space of actions, $\mathcal{A}$, that is, $\pi : \mathcal{S} \...
2
votes
1answer
51 views

Possible inconsistency in the Policy Improvement equation

I came across this formula in Sutton And Barto: RL an Intro (2nd Edition) equation number 4.7 (page number 78). If $\pi$ and $\pi'$ are deterministic policies and $q_\pi(s, \pi'(s)) \geq v_\pi(s)$ ...
2
votes
0answers
31 views

AlphaGo Zero: Does the policy head give a probability for every possible move?

If I understood correctly, the AlphaGo Zero network returns two values: a vector of logit probabilities p and a value v. My question is: in this vector that it is outputted, do we have a probability ...
1
vote
1answer
30 views

Do I need to store the policy for RL?

I am creating a zero-sum game with RL and wondered if I need to store the policy, or if there are other RL methods that produce similar results (consistently beating the human player) without the need ...
1
vote
1answer
176 views

What is the difference between a stationary and a non-stationary policy?

In reinforcement learning, there are deterministic and non-deterministic (or stochastic) policies, but there are also stationary and non-stationary policies. What is the difference between a ...
1
vote
0answers
22 views

Measure grid-world environments difference for reinforcement learning

I'd like to measure the difference between 2 grid-worlds to determine the generalization capacity of my agent using tabular Q-learning. Example (OpenAI Frozen Lake) : SFFF FHFH FFFH HFFG and : ...
1
vote
0answers
28 views

Where does the expectation term in the derivative of the soft-max policy come from?

At slide 17 of the David Silver's series, the soft-max policy is defined as follows $$ \pi_\theta(s, a) \propto e^{\phi(s, a)^T \theta} $$ that is, the probability of an action $a$ (in state $s$) is ...
0
votes
0answers
31 views

Finding optimal Value function and Policy for an MDP

I am solving an RL MDP problem which is model based. I have an MDP which has four possible states S1-S4 and four different actions A1-A4, with S4 being terminal state and S1 is the beginning state. ...
-1
votes
1answer
141 views

What is the difference between a stochastic and a deterministic policy?

In reinforcement learning, there are the concepts of stochastic (or probabilistic) and deterministic policies. What is the difference between them?
-1
votes
1answer
25 views

Does higher Accuracy in Reinforcement Learning indicate better model performance?

If a reinforcement learning algorithm uses a Deep Neural Network to predict the action given a state (a NN for a policy function), an Monte Carlo Tree Search in a model-based learning setup, then ...