# Questions tagged [policies]

For questions related to policies (as defined in reinforcement learning or other AI sub-fields).

60 questions
Filter by
Sorted by
Tagged with
140 views

### How can we find the value function by solving a system of linear equations without knowing the policy?

An MDP is a Markov Reward Process with decisions, it’s an environment in which all states are Markov. This is what we want to solve. An MDP is a tuple $(S, A, P, R, \gamma)$, where $S$ is our state ...
243 views

### Is my understanding of the value function, Q function, policy, reward and return correct?

I'm a beginner in the RL field, and I would like to check that my understanding of certain RL concepts. Value function: How good it is to be in a state S following policy π. ...
45 views

### In reinforcement learning, why are policies defined as functions of states and not observations?

I am new to RL and I am following Sutton & Barto's book. My doubt is, when we talk about the policy of our agent, we say it is the probability of taking some action $a$ given the state $s$. ...
50 views

The Sutton and Barto reinforcement learning textbook states that the value of a state under an optimal policy must equal the expected return for the best action from that state. That is, $$v_*(s) = \... 0answers 32 views ### How to allow RL systems to find better policies after code changes? Suppose that in version 1 of a reinforcement-learning system an optimal policy A got generated for executing a task. But, in a newer version 2 of that application (with new code changes), there ... 0answers 50 views ### Proof that there always exists a dominating policy in an MDP I think that it is common knowledge that for any infinite horizon discounted MDP (S, A, P, r, \gamma), there always exists a dominating policy \pi, i.e. a policy \pi such that for all policies \... 1answer 78 views ### Do we need multiple parallel environments to train in batches an on-policy algorithm? When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ... 3answers 5k views ### What is the difference between a stochastic and a deterministic policy? In reinforcement learning, there are the concepts of stochastic (or probabilistic) and deterministic policies. What is the difference between them? 0answers 27 views ### DQN learns to always choose the same action for all states I have created an RL model that uses QBased policy with a neural network for estimating Q values. My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. ... 1answer 51 views ### Is the policy gradient expression in Fundamentals of Deep Learning wrong? I don't understand the policy gradient as explained in Chapter-9 (Deep Reinforcement Learning) of the book Fundamentals of deep learning. Here is the whole paragraph: Policy Learning via Policy ... 1answer 44 views ### Is a learned policy, for a deterministic problem, trained in a supervised process, a stochastic policy? If I trained a neural network with 4 outputs (one for each action: move down, up, left, and right) to move an agent through a grid (deterministic problem). The output of the neural network is a ... 2answers 107 views ### Why is the optimal policy for an infinite horizon MDP deterministic? Could someone please help me gain some intuition as to why the optimal policy for a Markov Decision Process in the infinite horizon case (agent acts forever) is deterministic? 1answer 3k views ### What is the difference between a stationary and a non-stationary policy? In reinforcement learning, there are deterministic and non-deterministic (or stochastic) policies, but there are also stationary and non-stationary policies. What is the difference between a ... 2answers 107 views ### How does the AlphaGo Zero policy decide what move to execute? I was going through the AlphaGo Zero paper and I was trying to understand everything, but I just can't figure out this one formula:$$ \pi(a \mid s_0) = \frac{N(s_0, a)^{\frac{1}{\tau}}}{\sum_b N(s_0,...
68 views

I have implemented DQN algorithm and wonder why during testing, the best performance is achieved by a policy from about 300 episode, when mean Q values converge at about 800 episode? Mean Q-values ...
192 views

### What is a "learned policy" in Q-learning?

I am completing an assignment at the moment. One of the assignment questions asks how you identified the learned policy and how you obtained it. The question is a reinforcement learning question, and ...
3k views

### What does "stationary" mean in the context of reinforcement learning?

I think I've seen the expressions "stationary data", "stationary dynamics" and "stationary policy", among others, in the context of reinforcement learning. What does it mean? I think stationary policy ...
306 views

### Why does having a fixed policy change a Markov Decision Process to a Markov Reward Process?

If a policy is fixed, it is said that a Markov Decision Process (MDP) becomes a Markov Reward Process (MRP). Why is this so? Aren't the transitions and rewards still parameterized by the action and ...
27 views

70 views

### What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?

In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods. AlphaGo Deepmind RL model has beaten the best Go ...
131 views

59 views

### Why does the value of state change depending on the policy used to get to that state?

From what I understand, the value function estimates how 'good' it is for an agent to be in a state, and a policy is a mapping of actions to state. If I have understood these concepts correctly, why ...
44 views

### Off-policy full-random training in easy-to-explore environment

Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe). In those environments, using off-policy algorithm, is it a good ...
63 views

### Are there any reinforcement learning benchmarks where the optimal policy is known for each environment?

There are multiple reinforcement learning (RL) benchmarks (i.e. a set of environments where we can test our RL algorithms), for example, the DeepMind Control Suite. However, given that I am currently ...
132 views

### Why do we have two similar action selection strategies for UCB1?

In the literature, there are at least two action selection strategies associated with the UCB1's action selection strategy/policy. For example, in the paper Algorithms for the multi-armed bandit ...
35 views

I am training a network through reinforcement learning. The policy network learns rotations, but depending on the actual input (state), the output of the network should be restricted to be in certain ...
34 views

### What's an example of a simple policy but a complex value function?

Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared ...
58 views

### Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?

I was looking at the Bellman equation, and I noticed a difference between the equations used in policy evaluation and value iteration. In policy evaluation, there was the presence of $\pi(a \mid s)$, ...
26 views

### Do we assume the policy to be deterministic when proving the optimality?

In reinforcement learning, when we talk about the principle of optimality, do we assume the policy to be deterministic?
69 views

### Is a policy in reinforcement learning analogous to a field such as APF?

If a policy maps states to actions in reinforcement learning, then for a path planning with obstacles, can't we simply use Artificial Potential Field fields for path planning and model policy ...
67 views

### Is it common to have extreme policy's probabilities?

I have implemented several policy gradient algorithms (REINFORCE, A2C, and PPO) and am finding that the resultant policy's action probability distributions can be rather extreme. As a note, I have ...
121 views

### What is meant by "generate the data" in describing the difference between on-policy and off-policy?

From the book: Sutton, Richard S.,Barto, Andrew G.. Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 100). The MIT Press. Kindle Edition. " following is stated: "...
76 views

### Does the off-policy evaluation work for non-stationary policies?

As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies? For example, IS (importance sampling)-based estimators, such as weighted IS or doubly ...
209 views

### Why is the policy not a part of the MDP definition?

I'm reading an article on reinforcement learning, and I don't understand why the agent's policy $\pi$ is not part of definition of Markov Decision process(MDP): Bu, Lucian, Robert Babu, and Bart De ...
283 views

### Why do RL implementations converge on one action?

I have seen this happening in implementations of state-of-the-art RL algorithms where the model converges to a single action over time after multiple training iterations. Are there some general ...
65 views

### How to estimate a behavior policy for off-policy learning based on data?

I have a dataset which includes states, actions, and reward. The dataset includes information on the transition, i.e., $p(r,s' \mid s,a)$. Is there a way to estimate a behavior policy from this ...
79 views

### Why do I need an initial arbitrary policy to implement value iteration algorithm

I've been recently given an assignment based on Reinforcement Learning and I'm supposed to implement the value iteration algorithm in a grid environment. The assignment: My doubt is why do I even ...
513 views

### Why do value iteration and policy iteration obtain similar policies even though they have different value functions?

I am trying to implement value and policy iteration algorithms. My value function from policy iteration looks vastly different from the values from value iteration, but the policy obtained from both ...
60 views

### Why does the policy $\pi$ affect the Q value?

From my understanding, the policy $\pi$ is basically how the agent acts (i.e. the actions it will take in each state). However, I am confused about the Q value and how it is "affected" by a policy. ...
125 views

### How do I compute the variance of the return of an evaluation policy using two behaviour policies?

Suppose there is an evaluation policy called $\pi_{e}$ and there are two behavior policies $\pi_{b1}$ and $\pi_{b2}$. I know that it is possible to estimate the return of policy $\pi_{e}$ through ...
48 views

### Representation of state space, action space and reward system for Reinforcement Learning problem

I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without ...
71 views

### How to understand and visualize a trained RL agent's policy when the state space is high dimensional?

What are typical ways to understand and visualize a trained RL agent's policy when the state space is of high dimension (but not images)? For example, suppose state and action are denoted by \$s=(...
49 views

### Can the importance sampling estimator have a non-stationary behaviour policy even if the target policy is stationary?

The inverse propensity score (IPS) estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained in the paper Doubly Robust Policy Evaluation and Optimization. ...
79 views

### AlphaGo Zero: Does the policy head give a probability for every possible move?

If I understood correctly, the AlphaGo Zero network returns two values: a vector of logit probabilities p and a value v. My question is: in this vector that it is outputted, do we have a probability ...
120 views

### Why is having low variance important in offline policy evaluation of reinforcement learning?

Intuitively, I understand that having an unbiased estimate of a policy is important because being biased just means that our estimate is distant from the truth value. However, I don't understand ...