Newest 'policies' Questions

3

votes

1answer

41 views

In reinforcement learning, why are policies defined as functions of states and not observations?

I am new to RL and I am following Sutton & Barto's book. My doubt is, when we talk about the policy of our agent, we say it is the probability of taking some action $a$ given the state $s$. ...

asked Sep 19 at 4:44

hakiki_makato

1534 bronze badges

2

votes

1answer

50 views

Why must the value of a state under an optimal policy equal the expected return for the best action from that state?

The Sutton and Barto reinforcement learning textbook states that the value of a state under an optimal policy must equal the expected return for the best action from that state. That is, $$v_*(s) = \...

reinforcement-learning value-functions policies bellman-equations

asked Aug 22 at 0:21

bonzo_pippinpaddle

376 bronze badges

1

vote

0answers

32 views

How to allow RL systems to find better policies after code changes?

Suppose that in version 1 of a reinforcement-learning system an optimal policy $A$ got generated for executing a task. But, in a newer version 2 of that application (with new code changes), there ...

reinforcement-learning policies

asked Aug 9 at 10:33

user456280

1715 bronze badges

5

votes

0answers

50 views

Proof that there always exists a dominating policy in an MDP

I think that it is common knowledge that for any infinite horizon discounted MDP $(S, A, P, r, \gamma)$, there always exists a dominating policy $\pi$, i.e. a policy $\pi$ such that for all policies $\...

reinforcement-learning markov-decision-process proofs policies

asked Jul 16 at 16:21

Ben C.

1753 bronze badges

0

votes

0answers

27 views

DQN learns to always choose the same action for all states

I have created an RL model that uses QBased policy with a neural network for estimating Q values. My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. ...

reinforcement-learning deep-rl dqn policies

asked May 27 at 15:04

KnownUnknown

1

vote

1answer

51 views

Is the policy gradient expression in Fundamentals of Deep Learning wrong?

I don't understand the policy gradient as explained in Chapter-9 (Deep Reinforcement Learning) of the book Fundamentals of deep learning. Here is the whole paragraph: Policy Learning via Policy ...

deep-rl objective-functions math policy-gradients policies

asked Apr 8 at 8:04

Jonathan Sands

113 bronze badges

3

votes

1answer

130 views

How can we find the value function by solving a system of linear equations without knowing the policy?

An MDP is a Markov Reward Process with decisions, it’s an environment in which all states are Markov. This is what we want to solve. An MDP is a tuple $(S, A, P, R, \gamma)$, where $S$ is our state ...

markov-decision-process policies bellman-equations transition-model

asked Feb 16 at 8:51

Abc1729

353 bronze badges

2

votes

1answer

68 views

Why do I get the best policy before Q values converge using DQN?

I have implemented DQN algorithm and wonder why during testing, the best performance is achieved by a policy from about 300 episode, when mean Q values converge at about 800 episode? Mean Q-values ...

dqn deep-rl convergence value-functions policies

asked Feb 4 at 20:48

opal

314 bronze badges

2

votes

1answer

44 views

Is a learned policy, for a deterministic problem, trained in a supervised process, a stochastic policy?

If I trained a neural network with 4 outputs (one for each action: move down, up, left, and right) to move an agent through a grid (deterministic problem). The output of the neural network is a ...

neural-networks policies deterministic-policy stochastic-policy softmax-policy

asked Feb 3 at 12:47

Xtalker

212 bronze badges

2

votes

1answer

191 views

What is a "learned policy" in Q-learning?

I am completing an assignment at the moment. One of the assignment questions asks how you identified the learned policy and how you obtained it. The question is a reinforcement learning question, and ...

reinforcement-learning q-learning terminology value-functions policies

asked Jan 25 at 9:32

user43972

2

votes

0answers

27 views

Does there necessarily exist "dominated actions" in a MDP?

In a Markov Decision Process, is it possible that there exists no "dominated action"? I define a dominated action the following way: we say that $(s,a)$ is a dominated action, if $\forall \...

reinforcement-learning markov-decision-process proofs value-functions policies

asked Jan 9 at 19:27

Phil

313 bronze badges

1

vote

0answers

15 views

How to obtain the policy in the form of a finite-state controller from the value function vectors over the belief space of the POMDP?

I was reading this paper by Hansen. It says the following: A correspondence between vectors and one-step policy choices plays an important role in this interpretation of a policy. Each vector in $\...

papers value-functions policies pomdp policy-iteration

asked Dec 25 '20 at 8:41

Rnj

1815 bronze badges

1

vote

0answers

70 views

What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?

In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods. AlphaGo Deepmind RL model has beaten the best Go ...

reinforcement-learning policies model-based-methods model-free-methods value-based-methods

asked Dec 23 '20 at 6:14

user781486

1835 bronze badges

3

votes

1answer

131 views

Are there some notions of distance between two policies?

I want to determine some distance between two policies $\pi_1 (a \mid s)$ and $\pi_2 (a \mid s)$, i.e. something like $\vert \vert \pi_1 (a \mid s) - \pi_2(a \mid s) \vert \vert$, where $\pi_i (a\mid ...

reinforcement-learning reference-request policies kl-divergence wasserstein-metric

asked Dec 13 '20 at 16:20

Felix P.

2771 silver badge6 bronze badges

0

votes

1answer

59 views

How to use and update a shared/global policy between Reinforcement Learning Agents

I would be grateful for some guidance on a RL problem I am trying to solve where multiple RL agents use a common/global policy at the initial state of an episode in the RL Environment, and then update ...

reinforcement-learning environment policies

asked Nov 25 '20 at 12:14

RL_NOOB

153 bronze badges

5

votes

2answers

326 views

Given two optimal policies, is an affine combination of them also optimal?

If there are two different optimal policies $\pi_1, \pi_2$ in a reinforcement learning task, will the linear combination (or affine combination) of the two policies $\alpha \pi_1 + \beta \pi_2, \alpha ...

reinforcement-learning proofs policies optimal-policy optimality

asked Nov 18 '20 at 7:04

yang liu

533 bronze badges

0

votes

0answers

44 views

Off-policy full-random training in easy-to-explore environment

Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe). In those environments, using off-policy algorithm, is it a good ...

reinforcement-learning dqn policies off-policy-methods epsilon-greedy-policy

asked Nov 7 '20 at 17:59

Loheek

2561 silver badge5 bronze badges

0

votes

0answers

63 views

Are there any reinforcement learning benchmarks where the optimal policy is known for each environment?

There are multiple reinforcement learning (RL) benchmarks (i.e. a set of environments where we can test our RL algorithms), for example, the DeepMind Control Suite. However, given that I am currently ...

reinforcement-learning reference-request policies benchmarks

asked Oct 29 '20 at 11:13

nbro♦

29k7 gold badges59 silver badges122 bronze badges

3

votes

1answer

132 views

Why do we have two similar action selection strategies for UCB1?

In the literature, there are at least two action selection strategies associated with the UCB1's action selection strategy/policy. For example, in the paper Algorithms for the multi-armed bandit ...

reinforcement-learning policies sutton-barto multi-armed-bandit upper-confidence-bound

asked Oct 23 '20 at 10:32

nbro♦

29k7 gold badges59 silver badges122 bronze badges

1

vote

0answers

35 views

Dynamically adapting activation function

I am training a network through reinforcement learning. The policy network learns rotations, but depending on the actual input (state), the output of the network should be restricted to be in certain ...

neural-networks reinforcement-learning activation-function policies state-of-the-art

asked Oct 19 '20 at 22:12

Ardi

312 bronze badges

2

votes

0answers

34 views

What's an example of a simple policy but a complex value function?

Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared ...

reinforcement-learning policy-gradients policies value-functions real-world

asked Aug 27 '20 at 10:16

epsilon-emperor

8901 silver badge13 bronze badges

2

votes

1answer

58 views

Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?

I was looking at the Bellman equation, and I noticed a difference between the equations used in policy evaluation and value iteration. In policy evaluation, there was the presence of $\pi(a \mid s)$, ...

reinforcement-learning policies value-iteration policy-iteration bellman-equations

asked Aug 25 '20 at 12:35

Chukwudi Ogbonna

954 bronze badges

2

votes

0answers

26 views

Do we assume the policy to be deterministic when proving the optimality?

In reinforcement learning, when we talk about the principle of optimality, do we assume the policy to be deterministic?

reinforcement-learning proofs policies deterministic-policy

asked Aug 18 '20 at 9:32

hakiki_makato

1534 bronze badges

1

vote

2answers

107 views

Why is the optimal policy for an infinite horizon MDP deterministic?

Could someone please help me gain some intuition as to why the optimal policy for a Markov Decision Process in the infinite horizon case (agent acts forever) is deterministic?

reinforcement-learning markov-decision-process policies optimal-policy optimality

asked Aug 6 '20 at 7:42

epsilon-emperor

8901 silver badge13 bronze badges

2

votes

1answer

69 views

Is a policy in reinforcement learning analogous to a field such as APF?

If a policy maps states to actions in reinforcement learning, then for a path planning with obstacles, can't we simply use Artificial Potential Field fields for path planning and model policy ...

reinforcement-learning markov-decision-process policies path-planning

asked Jul 28 '20 at 15:19

gfdsal

1616 bronze badges

2

votes

1answer

67 views

Is it common to have extreme policy's probabilities?

I have implemented several policy gradient algorithms (REINFORCE, A2C, and PPO) and am finding that the resultant policy's action probability distributions can be rather extreme. As a note, I have ...

reinforcement-learning policy-gradients policies

asked Jul 20 '20 at 21:11

curiouscat22

333 bronze badges

3

votes

1answer

121 views

What is meant by "generate the data" in describing the difference between on-policy and off-policy?

From the book: Sutton, Richard S.,Barto, Andrew G.. Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 100). The MIT Press. Kindle Edition. " following is stated: "...

reinforcement-learning policies off-policy-methods

asked Jul 12 '20 at 14:15

blue-sky

2791 silver badge9 bronze badges

1

vote

1answer

78 views

Do we need multiple parallel environments to train in batches an on-policy algorithm?

When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ...

reinforcement-learning environment policies experience-replay on-policy-methods

asked Jul 2 '20 at 19:46

Daniel

1112 bronze badges

1

vote

0answers

76 views

Does the off-policy evaluation work for non-stationary policies?

As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies? For example, IS (importance sampling)-based estimators, such as weighted IS or doubly ...

reinforcement-learning policies off-policy-methods importance-sampling

asked Jun 21 '20 at 4:21

Hunnam

2071 silver badge5 bronze badges

2

votes

2answers

208 views

Why is the policy not a part of the MDP definition?

I'm reading an article on reinforcement learning, and I don't understand why the agent's policy $\pi$ is not part of definition of Markov Decision process(MDP): Bu, Lucian, Robert Babu, and Bart De ...

reinforcement-learning definitions markov-decision-process policies

asked May 23 '20 at 19:03

Raphael Augusto

1877 bronze badges

3

votes

1answer

280 views

Why do RL implementations converge on one action?

I have seen this happening in implementations of state-of-the-art RL algorithms where the model converges to a single action over time after multiple training iterations. Are there some general ...

reinforcement-learning convergence policies

asked May 10 '20 at 15:39

JAYDEEP GHOSE

511 bronze badge

3

votes

2answers

107 views

How does the AlphaGo Zero policy decide what move to execute?

I was going through the AlphaGo Zero paper and I was trying to understand everything, but I just can't figure out this one formula: $$ \pi(a \mid s_0) = \frac{N(s_0, a)^{\frac{1}{\tau}}}{\sum_b N(s_0,...

reinforcement-learning policies deepmind alphago-zero alphago

asked May 7 '20 at 11:09

Eloi M.

334 bronze badges

2

votes

3answers

65 views

How to estimate a behavior policy for off-policy learning based on data?

I have a dataset which includes states, actions, and reward. The dataset includes information on the transition, i.e., $p(r,s' \mid s,a)$. Is there a way to estimate a behavior policy from this ...

reinforcement-learning policies off-policy-methods

asked Apr 27 '20 at 14:44

Tracy Yang

2712 silver badges4 bronze badges

3

votes

1answer

506 views

Why do value iteration and policy iteration obtain similar policies even though they have different value functions?

I am trying to implement value and policy iteration algorithms. My value function from policy iteration looks vastly different from the values from value iteration, but the policy obtained from both ...

reinforcement-learning comparison policies value-iteration policy-iteration

asked Apr 21 '20 at 20:03

PyWalker27

1851 silver badge6 bronze badges

2

votes

1answer

79 views

Why do I need an initial arbitrary policy to implement value iteration algorithm

I've been recently given an assignment based on Reinforcement Learning and I'm supposed to implement the value iteration algorithm in a grid environment. The assignment: My doubt is why do I even ...

reinforcement-learning policies value-functions value-iteration

asked Apr 16 '20 at 12:11

Carrick

1215 bronze badges

2

votes

2answers

240 views

Is my understanding of the value function, Q function, policy, reward and return correct?

I'm a beginner in the RL field, and I would like to check that my understanding of certain RL concepts. Value function: How good it is to be in a state S following policy π. ...

reinforcement-learning q-learning terminology policies return

asked Apr 16 '20 at 1:53

BG10

1156 bronze badges

1

vote

2answers

60 views

Why does the policy $\pi$ affect the Q value?

From my understanding, the policy $\pi$ is basically how the agent acts (i.e. the actions it will take in each state). However, I am confused about the Q value and how it is "affected" by a policy. ...

reinforcement-learning q-learning policies value-functions

asked Apr 16 '20 at 0:52

BG10

1156 bronze badges

3

votes

0answers

48 views

Representation of state space, action space and reward system for Reinforcement Learning problem

I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without ...

reinforcement-learning dqn policy-gradients policies

asked Mar 22 '20 at 20:22

Incompleteness

312 bronze badges

0

votes

0answers

71 views

How to understand and visualize a trained RL agent's policy when the state space is high dimensional?

What are typical ways to understand and visualize a trained RL agent's policy when the state space is of high dimension (but not images)? For example, suppose state and action are denoted by $s=(...

reinforcement-learning markov-decision-process policies

asked Mar 3 '20 at 16:39

Ryan

1214 bronze badges

2

votes

0answers

49 views

Can the importance sampling estimator have a non-stationary behaviour policy even if the target policy is stationary?

The inverse propensity score (IPS) estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained in the paper Doubly Robust Policy Evaluation and Optimization. ...

reinforcement-learning comparison policies off-policy-methods importance-sampling

asked Jan 21 '20 at 19:53

Hunnam

2071 silver badge5 bronze badges

2

votes

0answers

79 views

AlphaGo Zero: Does the policy head give a probability for every possible move?

If I understood correctly, the AlphaGo Zero network returns two values: a vector of logit probabilities p and a value v. My question is: in this vector that it is outputted, do we have a probability ...

deep-learning policies deepmind alphago-zero

asked Nov 11 '19 at 1:33

ihavenoidea

2251 silver badge8 bronze badges

3

votes

1answer

306 views

Why does having a fixed policy change a Markov Decision Process to a Markov Reward Process?

If a policy is fixed, it is said that a Markov Decision Process (MDP) becomes a Markov Reward Process (MRP). Why is this so? Aren't the transitions and rewards still parameterized by the action and ...

reinforcement-learning markov-decision-process policies markov-reward-process

asked Sep 30 '19 at 2:15

Peter

333 bronze badges

4

votes

2answers

120 views

Why is having low variance important in offline policy evaluation of reinforcement learning?

Intuitively, I understand that having an unbiased estimate of a policy is important because being biased just means that our estimate is distant from the truth value. However, I don't understand ...

reinforcement-learning policies bias-variance-tradeoff

asked Sep 29 '19 at 15:36

Hunnam

2071 silver badge5 bronze badges

3

votes

1answer

177 views

Can someone please help me validate my MDP?

Problem Statement : I have a system with four states - S1 through S4 where S1 is the beginning state and S4 is the end/terminal state. The next state is always better than the previous state i.e if ...

reinforcement-learning rewards policies markov-decision-process

asked Sep 5 '19 at 15:08

Bhavana

736 bronze badges

1

vote

0answers

50 views

Finding optimal Value function and Policy for an MDP

I am solving an RL MDP problem which is model based. I have an MDP which has four possible states S1-S4 and four different actions A1-A4, with S4 being terminal state and S1 is the beginning state. ...

reinforcement-learning rewards policies markov-decision-process

asked Sep 4 '19 at 10:52

Bhavana

736 bronze badges

1

vote

1answer

36 views

Do I need to store the policy for RL?

I am creating a zero-sum game with RL and wondered if I need to store the policy, or if there are other RL methods that produce similar results (consistently beating the human player) without the need ...

reinforcement-learning policies off-policy-methods storage

asked Aug 10 '19 at 16:06

mason7663

5732 silver badges7 bronze badges

2

votes

1answer

48 views

If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy?

I wrote a simple implementation of Flappy Bird in Python, and now I'm trying to train an agent to play it at a reasonable skill level using TFLearn. I feed the network an input vector of size 4: ...

q-learning deep-rl policies

asked Jul 13 '19 at 21:48

sOvr9000

336 bronze badges

6

votes

1answer

3k views

What is the difference between a stationary and a non-stationary policy?

In reinforcement learning, there are deterministic and non-deterministic (or stochastic) policies, but there are also stationary and non-stationary policies. What is the difference between a ...

reinforcement-learning comparison policies stationary-policy

asked Jun 27 '19 at 15:14

nbro♦

29k7 gold badges59 silver badges122 bronze badges

4

votes

3answers

5k views

What is the difference between a stochastic and a deterministic policy?

In reinforcement learning, there are the concepts of stochastic (or probabilistic) and deterministic policies. What is the difference between them?

reinforcement-learning comparison policies deterministic-policy stochastic-policy

asked May 12 '19 at 18:50

nbro♦

29k7 gold badges59 silver badges122 bronze badges

2

votes

1answer

72 views

Possible inconsistency in the Policy Improvement equation

I came across this formula in Sutton And Barto: RL an Intro (2nd Edition) equation number 4.7 (page number 78). If $\pi$ and $\pi'$ are deterministic policies and $q_\pi(s, \pi'(s)) \geq v_\pi(s)$ ...

reinforcement-learning sutton-barto policies

asked May 11 '19 at 6:38

user9947

Questions tagged [policies]

Related Tags