# Questions tagged [policies]

For questions related to policies (as defined in reinforcement learning or other AI sub-fields).

79 questions
Filter by
Sorted by
Tagged with
40 views

### Are there problems where the optimal policy is stochastic?

I know that in a MDP there always exists a unique optimal deterministic policy. Does a statement like this also exist for optimal stochastic policies? Is there also always a unique optimal stochastic ...
• 113
41 views

### Policy in on-policy algorithms and experience replay

For SARSA algorithm, assuming that we initialize all $Q(s,a)$ to $0$, then in the first iteration, all actions are the best actions as $Q$ values are the same ($0$). So the behavior policy in this ...
• 55
1 vote
42 views

### Why is the better policy defined with respect to all the states values being greater?

In Sutton & Barto (Section 3.6 - Optimal Policies and Optimal Value Functions), they say that : Value functions define a partial ordering over policies. A policy $\pi$ is defined to be better ...
• 13
8 views

### What is the name of this construction for a compound policy that operates over distinct action sets?

I am developing an RL algorithm with a policy that needs to compute valid probabilities over multiple distinct action sets. I think I have a construction that will work, but I do not know what it is ...
67 views

### What happens when the probability of either one of the policies is 0 in Importance Sampling?

I have a general question about the methods that use importance sampling in RL. What happens when the probability of either one of the policies is 0?
• 21
368 views

### Is my understanding correct regarding the difference between policy and plan?

I am confused regarding the difference between policy and plan in reinforcement learning. According to my understanding, when we calculate the value of state using Bellman equation in deterministic ...
• 111
1 vote
101 views

### What is the best distance measure between policies that are not probability distributions?

This question asks if there is a way to measure distance between policies that are in fact probability distributions. In the case of continuous control with deterministic policies where they take a ...
30 views

### Define possible?

In Reinforcement Learning, policies are defined in terms of possible actions (see for instance page 58 of the book by Sutton et al.). So, is any action that an agent has in its repertoire always "...
83 views

### Policy and Discount Factor

This question is similar to this question, however it has a different question. I'm learning MDP's and I'd like to know if I'm doing these exercises correctly: Consider the following MDP: Suppose a ...
• 145
1 vote
451 views

### If a policy is epsilon-greedy, is it technically stochastic?

Even though if exploration doesn't happen, it's deterministic.
• 93
334 views

• 133
253 views

### In reinforcement learning, why are policies defined as functions of states and not observations?

I am new to RL and I am following Sutton & Barto's book. My doubt is, when we talk about the policy of our agent, we say it is the probability of taking some action $a$ given the state $s$. ...
412 views

The Sutton and Barto reinforcement learning textbook states that the value of a state under an optimal policy must equal the expected return for the best action from that state. That is, $$v_*(s) = \... 1 vote 0 answers 34 views ### How to allow RL systems to find better policies after code changes? Suppose that in version 1 of a reinforcement-learning system an optimal policy A got generated for executing a task. But, in a newer version 2 of that application (with new code changes), there ... 6 votes 1 answer 118 views ### Proof that there always exists a dominating policy in an MDP I think that it is common knowledge that for any infinite horizon discounted MDP (S, A, P, r, \gamma), there always exists a dominating policy \pi, i.e. a policy \pi such that for all policies \... • 185 0 votes 0 answers 376 views ### DQN learns to always choose the same action for all states I have created an RL model that uses QBased policy with a neural network for estimating Q values. My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. ... 1 vote 1 answer 900 views ### Are optimal policies always deterministic, or can there also be optimal policies that are stochastic? Let M be an MDP with two states, A and B, where A is the starting state, and you always transit to the final state B using two possible actions. A_1 gives you rewards that are normally ... • 45 1 vote 1 answer 150 views ### Is the policy gradient expression in Fundamentals of Deep Learning wrong? I don't understand the policy gradient as explained in Chapter-9 (Deep Reinforcement Learning) of the book Fundamentals of deep learning. Here is the whole paragraph: Policy Learning via Policy ... 3 votes 1 answer 355 views ### How can we find the value function by solving a system of linear equations without knowing the policy? An MDP is a Markov Reward Process with decisions, it’s an environment in which all states are Markov. This is what we want to solve. An MDP is a tuple (S, A, P, R, \gamma), where S is our state ... • 45 2 votes 1 answer 117 views ### Why do I get the best policy before Q values converge using DQN? I have implemented DQN algorithm and wonder why during testing, the best performance is achieved by a policy from about 300 episode, when mean Q values converge at about 800 episode? Mean Q-values ... 2 votes 1 answer 106 views ### Is a learned policy, for a deterministic problem, trained in a supervised process, a stochastic policy? If I trained a neural network with 4 outputs (one for each action: move down, up, left, and right) to move an agent through a grid (deterministic problem). The output of the neural network is a ... • 21 3 votes 1 answer 1k views ### What is a "learned policy" in Q-learning? I am completing an assignment at the moment. One of the assignment questions asks how you identified the learned policy and how you obtained it. The question is a reinforcement learning question, and ... 2 votes 0 answers 55 views ### Does there necessarily exist "dominated actions" in a MDP? In a Markov Decision Process, is it possible that there exists no "dominated action"? I define a dominated action the following way: we say that (s,a) is a dominated action, if \forall \... • 31 1 vote 0 answers 19 views ### How to obtain the policy in the form of a finite-state controller from the value function vectors over the belief space of the POMDP? I was reading this paper by Hansen. It says the following: A correspondence between vectors and one-step policy choices plays an important role in this interpretation of a policy. Each vector in \... • 221 2 votes 0 answers 159 views ### What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player? In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods. AlphaGo Deepmind RL model has beaten the best Go ... • 201 3 votes 1 answer 626 views ### Are there some notions of distance between two policies? I want to determine some distance between two policies \pi_1 (a \mid s) and \pi_2 (a \mid s), i.e. something like \vert \vert \pi_1 (a \mid s) - \pi_2(a \mid s) \vert \vert, where \pi_i (a\mid ... • 297 0 votes 1 answer 249 views ### How to use and update a shared/global policy between Reinforcement Learning Agents I would be grateful for some guidance on a RL problem I am trying to solve where multiple RL agents use a common/global policy at the initial state of an episode in the RL Environment, and then update ... • 15 5 votes 2 answers 1k views ### Given two optimal policies, is an affine combination of them also optimal? If there are two different optimal policies \pi_1, \pi_2 in a reinforcement learning task, will the linear combination (or affine combination) of the two policies \alpha \pi_1 + \beta \pi_2, \alpha ... • 53 0 votes 0 answers 53 views ### Off-policy full-random training in easy-to-explore environment Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe). In those environments, using off-policy algorithm, is it a good ... • 266 4 votes 1 answer 574 views ### Why do we have two similar action selection strategies for UCB1? In the literature, there are at least two action selection strategies associated with the UCB1's action selection strategy/policy. For example, in the paper Algorithms for the multi-armed bandit ... • 40.9k 1 vote 0 answers 92 views ### Dynamically adapting activation function I am training a network through reinforcement learning. The policy network learns rotations, but depending on the actual input (state), the output of the network should be restricted to be in certain ... • 31 2 votes 0 answers 54 views ### What are some other real-life examples of simple policies but complex value functions? Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared ... • 1,161 4 votes 1 answer 185 views ### Why doesn't value iteration use \pi(a \mid s) while policy evaluation does? I was looking at the Bellman equation, and I noticed a difference between the equations used in policy evaluation and value iteration. In policy evaluation, there was the presence of \pi(a \mid s), ... 2 votes 0 answers 31 views ### Do we assume the policy to be deterministic when proving the optimality? In reinforcement learning, when we talk about the principle of optimality, do we assume the policy to be deterministic? 2 votes 2 answers 808 views ### Why is the optimal policy for an infinite horizon MDP deterministic? Could someone please help me gain some intuition as to why the optimal policy for a Markov Decision Process in the infinite horizon case (agent acts forever) is deterministic? • 1,161 2 votes 1 answer 75 views ### Is a policy in reinforcement learning analogous to a field such as APF? If a policy maps states to actions in reinforcement learning, then for a path planning with obstacles, can't we simply use Artificial Potential Field fields for path planning and model policy ... • 171 2 votes 1 answer 94 views ### Is it common to have extreme policy's probabilities? I have implemented several policy gradient algorithms (REINFORCE, A2C, and PPO) and am finding that the resultant policy's action probability distributions can be rather extreme. As a note, I have ... 3 votes 1 answer 158 views ### What is meant by "generate the data" in describing the difference between on-policy and off-policy? From the book: Sutton, Richard S.,Barto, Andrew G.. Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 100). The MIT Press. Kindle Edition. " following is stated: "... • 335 1 vote 1 answer 325 views ### Do we need multiple parallel environments to train in batches an on-policy algorithm? When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ... • 111 2 votes 0 answers 116 views ### Does the off-policy evaluation work for non-stationary policies? As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies? For example, IS (importance sampling)-based estimators, such as weighted IS or doubly ... • 227 2 votes 1 answer 158 views ### Why is there an inconsistency between my calculations of Policy Iteration and this Sutton & Barto's diagram? In equation 4.9 of Sutton and Barto's book on page 79, we have (for the policy iteration algorithm):$$\pi'(s) = arg \max_{a}\sum_{s',r}p(s',r|s,a)[r+\gamma v_{\pi}(s')]$$where \pi is the previous ... • 147 2 votes 3 answers 361 views ### Why is the policy not a part of the MDP definition? I'm reading an article on reinforcement learning, and I don't understand why the agent's policy \pi is not part of definition of Markov Decision process(MDP): Bu, Lucian, Robert Babu, and Bart De ... 3 votes 1 answer 2k views ### Why do RL implementations converge on one action? I have seen this happening in implementations of state-of-the-art RL algorithms where the model converges to a single action over time after multiple training iterations. Are there some general ... 3 votes 2 answers 242 views ### How does the AlphaGo Zero policy decide what move to execute? I was going through the AlphaGo Zero paper and I was trying to understand everything, but I just can't figure out this one formula:$$ \pi(a \mid s_0) = \frac{N(s_0, a)^{\frac{1}{\tau}}}{\sum_b N(s_0,...
• 33
I have a dataset which includes states, actions, and reward. The dataset includes information on the transition, i.e., $p(r,s' \mid s,a)$. Is there a way to estimate a behavior policy from this ...