Questions tagged [policies]
For questions related to policies (as defined in reinforcement learning or other AI sub-fields).
76
questions
2
votes
2
answers
58
views
What happens when the probability of either one of the policies is 0 in Importance Sampling?
I have a general question about the methods that use importance sampling in RL. What happens when the probability of either one of the policies is 0?
- 39.7k
1
vote
1
answer
88
views
What is the best distance measure between policies that are not probability distributions?
This question asks if there is a way to measure distance between policies that are in fact probability distributions.
In the case of continuous control with deterministic policies where they take a ...
CommunityBot
- 1
2
votes
3
answers
329
views
Why is the policy not a part of the MDP definition?
I'm reading an article on reinforcement learning, and I don't understand why the agent's policy $\pi$ is not part of definition of Markov Decision process(MDP):
Bu, Lucian, Robert Babu, and Bart De ...
0
votes
0
answers
14
views
Finding an optimal action score function for Multi-Armed Bandit Problem
Considering a multi-armed bandit problem where there are :
...
-1
votes
1
answer
236
views
Is my understanding correct regarding the difference between policy and plan?
I am confused regarding the difference between policy and plan in reinforcement learning. According to my understanding, when we calculate the value of state using Bellman equation in deterministic ...
- 39.7k
0
votes
0
answers
30
views
Define possible?
In Reinforcement Learning, policies are defined in terms of possible actions (see for instance page 58 of the book by Sutton et al.). So, is any action that an agent has in its repertoire always "...
0
votes
0
answers
75
views
Policy and Discount Factor
This question is similar to this question, however it has a different question.
I'm learning MDP's and I'd like to know if I'm doing these exercises correctly:
Consider the following MDP:
Suppose a ...
- 145
1
vote
3
answers
356
views
If a policy is epsilon-greedy, is it technically stochastic?
Even though if exploration doesn't happen, it's deterministic.
- 823
0
votes
1
answer
234
views
What is the equation for $\pi_*$ in terms of $q_*(s,a)$?
I am trying to solve the following exercise from Sutton and Barto:
Sutton and Barto Exercise 3.27 Give an equation for $\pi_*$ in terms of $q_*(s,a)$
However, I am struggling to do so. I know that $\...
- 39.7k
2
votes
1
answer
738
views
Is Q-learning only capable of learning a deterministic policy?
I was following a reinforcement learning course on coursera and in this video at 2:57 the instructor says
Expected SARSA and SARSA both allow us to learn an optimal
$\epsilon$-soft policy, but, Q-...
- 39.7k
1
vote
2
answers
929
views
What is the difference between a policy and rewards?
I don't understand the difference between a policy and rewards. Sure, a policy tells us what to do, but isn't the output of a neural network trained on rewards basically a policy (i.e. choose the ...
- 39.7k
2
votes
1
answer
322
views
Is the initialisation of $V(s)$ and $\pi(s)$ really arbitrary in policy iteration?
In Sutton and Barto's book (Reinforcement learning: An introduction. MIT press, 2018), the algorithm "Policy Iteration" is:
Here, $V(s)$ is initialized arbitrarily, meaning that I can ...
CommunityBot
- 1
2
votes
1
answer
156
views
Why is there an inconsistency between my calculations of Policy Iteration and this Sutton & Barto's diagram?
In equation 4.9 of Sutton and Barto's book on page 79, we have (for the policy iteration algorithm):
$$\pi'(s) = arg \max_{a}\sum_{s',r}p(s',r|s,a)[r+\gamma v_{\pi}(s')]$$
where $\pi$ is the previous ...
- 39.7k
0
votes
1
answer
462
views
What's the benefit of repeating an action for a consecutive number of time steps?
What's the benefit of repeating an action for a consecutive number of time steps? Is there a way to tell if an agent in a given environment might perform better from repeated actions?
I came across an ...
- 829
1
vote
1
answer
299
views
Do we need multiple parallel environments to train in batches an on-policy algorithm?
When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ...
CommunityBot
- 1
2
votes
0
answers
52
views
What are some other real-life examples of simple policies but complex value functions?
Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared ...
- 39.7k
0
votes
0
answers
348
views
DQN learns to always choose the same action for all states
I have created an RL model that uses QBased policy with a neural network for estimating Q values.
My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. ...
1
vote
1
answer
664
views
Are optimal policies always deterministic, or can there also be optimal policies that are stochastic?
Let $M$ be an MDP with two states, $A$ and $B$, where $A$ is the starting state, and you always transit to the final state $B$ using two possible actions. $A_1$ gives you rewards that are normally ...
- 39.7k
5
votes
2
answers
618
views
Why is the derivative of this objective function 0 if the policy is deterministic?
In the Berkeley RL class CS294-112 Fa18 9/5/18, they mention the following gradient would be 0 if the policy is deterministic.
$$
\nabla_{\theta} J(\theta)=E_{\tau \sim \pi_{\theta}(\tau)}\left[\left(\...
- 39.7k
3
votes
3
answers
943
views
Is the policy really invariant under affine transformations of the reward function?
In the context of a Markov decision process, this paper says
it is well-known that the optimal policy is invariant to positive affine transformation of the reward function
On the other hand, ...
6
votes
1
answer
549
views
What is the relation between a policy which is the solution to a MDP and a policy like $\epsilon$-greedy?
In the context of reinforcement learning, a policy, $\pi$, is often defined as a function from the space of states, $\mathcal{S}$, to the space of actions, $\mathcal{A}$, that is, $\pi : \mathcal{S} \...
- 39.7k
3
votes
1
answer
339
views
Can an optimal policy have a value function that has a smaller value for a state than a non-optimal policy?
I'm starting to learn about the Bellman Equation and a question came to my mind.
A policy $\pi$ is optimal if the value $v_\pi(s)$ is greater or equal than the value $v_{\pi'}(s)$ for all states $s \...
- 39.7k
3
votes
2
answers
2k
views
Why do value iteration and policy iteration obtain similar policies even though they have different value functions?
I am trying to implement value and policy iteration algorithms. My value function from policy iteration looks vastly different from the values from value iteration, but the policy obtained from both ...
CommunityBot
- 1
3
votes
1
answer
327
views
How can we find the value function by solving a system of linear equations without knowing the policy?
An MDP is a Markov Reward Process with decisions, it’s an environment in which all states are Markov. This is what we want to solve. An MDP is a tuple $(S, A, P, R, \gamma)$, where $S$ is our state ...
- 30.5k
2
votes
2
answers
862
views
Is my understanding of the value function, Q function, policy, reward and return correct?
I'm a beginner in the RL field, and I would like to check that my understanding of certain RL concepts.
Value function: How good it is to be in a state S following policy π.
...
- 1
3
votes
1
answer
149
views
In reinforcement learning, why are policies defined as functions of states and not observations?
I am new to RL and I am following Sutton & Barto's book.
My doubt is, when we talk about the policy of our agent, we say it is the probability of taking some action $a$ given the state $s$. ...
- 580
3
votes
1
answer
303
views
Why must the value of a state under an optimal policy equal the expected return for the best action from that state?
The Sutton and Barto reinforcement learning textbook states that
the value of a state under an optimal policy must equal the expected return for the best action from that state.
That is,
$$v_*(s) = \...
1
vote
0
answers
33
views
How to allow RL systems to find better policies after code changes?
Suppose that in version 1 of a reinforcement-learning system an optimal policy $A$ got generated for executing a task. But, in a newer version 2 of that application (with new code changes), there ...
- 39.7k
6
votes
0
answers
99
views
Proof that there always exists a dominating policy in an MDP
I think that it is common knowledge that for any infinite horizon discounted MDP $(S, A, P, r, \gamma)$, there always exists a dominating policy $\pi$, i.e. a policy $\pi$ such that for all policies $\...
- 39.7k
8
votes
3
answers
16k
views
What is the difference between a stochastic and a deterministic policy?
In reinforcement learning, there are the concepts of stochastic (or probabilistic) and deterministic policies. What is the difference between them?
- 39.7k
1
vote
1
answer
135
views
Is the policy gradient expression in Fundamentals of Deep Learning wrong?
I don't understand the policy gradient as explained in Chapter-9 (Deep Reinforcement Learning) of the book Fundamentals of deep learning.
Here is the whole paragraph:
Policy Learning via Policy ...
2
votes
1
answer
104
views
Is a learned policy, for a deterministic problem, trained in a supervised process, a stochastic policy?
If I trained a neural network with 4 outputs (one for each action: move down, up, left, and right) to move an agent through a grid (deterministic problem). The output of the neural network is a ...
- 1,094
1
vote
2
answers
651
views
Why is the optimal policy for an infinite horizon MDP deterministic?
Could someone please help me gain some intuition as to why the optimal policy for a Markov Decision Process in the infinite horizon case (agent acts forever) is deterministic?
- 39.7k
8
votes
1
answer
7k
views
What is the difference between a stationary and a non-stationary policy?
In reinforcement learning, there are deterministic and non-deterministic (or stochastic) policies, but there are also stationary and non-stationary policies.
What is the difference between a ...
- 4,685
3
votes
2
answers
214
views
How does the AlphaGo Zero policy decide what move to execute?
I was going through the AlphaGo Zero paper and I was trying to understand everything, but I just can't figure out this one formula:
$$
\pi(a \mid s_0) = \frac{N(s_0, a)^{\frac{1}{\tau}}}{\sum_b N(s_0,...
2
votes
1
answer
106
views
Why do I get the best policy before Q values converge using DQN?
I have implemented DQN algorithm and wonder why during testing, the best performance is achieved by a policy from about 300 episode, when mean Q values converge at about 800 episode?
Mean Q-values ...
- 39.7k
3
votes
1
answer
1k
views
What is a "learned policy" in Q-learning?
I am completing an assignment at the moment. One of the assignment questions asks how you identified the learned policy and how you obtained it. The question is a reinforcement learning question, and ...
- 39.7k
15
votes
4
answers
7k
views
What does "stationary" mean in the context of reinforcement learning?
I think I've seen the expressions "stationary data", "stationary dynamics" and "stationary policy", among others, in the context of reinforcement learning. What does it mean? I think stationary policy ...
- 10.1k
4
votes
1
answer
741
views
Why does having a fixed policy change a Markov Decision Process to a Markov Reward Process?
If a policy is fixed, it is said that a Markov Decision Process (MDP) becomes a Markov Reward Process (MRP).
Why is this so? Aren't the transitions and rewards still parameterized by the action and ...
- 39.7k
2
votes
0
answers
54
views
Does there necessarily exist "dominated actions" in a MDP?
In a Markov Decision Process, is it possible that there exists no "dominated action"?
I define a dominated action the following way:
we say that $(s,a)$ is a dominated action, if $\forall \...
- 31
1
vote
0
answers
19
views
How to obtain the policy in the form of a finite-state controller from the value function vectors over the belief space of the POMDP?
I was reading this paper by Hansen.
It says the following:
A correspondence between vectors and one-step policy choices plays an important role in this interpretation of a policy. Each vector in $\...
- 39.7k
2
votes
0
answers
103
views
What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?
In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods.
AlphaGo Deepmind RL model has beaten the best Go ...
- 79
3
votes
1
answer
555
views
Are there some notions of distance between two policies?
I want to determine some distance between two policies $\pi_1 (a \mid s)$ and $\pi_2 (a \mid s)$, i.e. something like $\vert \vert \pi_1 (a \mid s) - \pi_2(a \mid s) \vert \vert$, where $\pi_i (a\mid ...
- 39.7k
0
votes
1
answer
215
views
How to use and update a shared/global policy between Reinforcement Learning Agents
I would be grateful for some guidance on a RL problem I am trying to solve where multiple RL agents use a common/global policy at the initial state of an episode in the RL Environment, and then update ...
- 151
5
votes
2
answers
1k
views
Given two optimal policies, is an affine combination of them also optimal?
If there are two different optimal policies $\pi_1, \pi_2$ in a reinforcement learning task, will the linear combination (or affine combination) of the two policies $\alpha \pi_1 + \beta \pi_2, \alpha ...
- 39.7k
1
vote
1
answer
153
views
Why does the value of state change depending on the policy used to get to that state?
From what I understand, the value function estimates how 'good' it is for an agent to be in a state, and a policy is a mapping of actions to state.
If I have understood these concepts correctly, why ...
- 39.7k
0
votes
0
answers
53
views
Off-policy full-random training in easy-to-explore environment
Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe).
In those environments, using off-policy algorithm, is it a good ...
- 266
4
votes
1
answer
449
views
Why do we have two similar action selection strategies for UCB1?
In the literature, there are at least two action selection strategies associated with the UCB1's action selection strategy/policy. For example, in the paper Algorithms for the multi-armed bandit ...
- 21
1
vote
0
answers
87
views
Dynamically adapting activation function
I am training a network through reinforcement learning. The policy network learns rotations, but depending on the actual input (state), the output of the network should be restricted to be in certain ...
- 31
4
votes
1
answer
145
views
Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?
I was looking at the Bellman equation, and I noticed a difference between the equations used in policy evaluation and value iteration.
In policy evaluation, there was the presence of $\pi(a \mid s)$, ...
- 39.7k