Questions tagged [policies]
For questions related to policies (as defined in reinforcement learning or other AI sub-fields).
53
questions
3
votes
1answer
115 views
How can we find the value function by solving a system of linear equations without knowing the policy?
An MDP is a Markov Reward Process with decisions, it’s an environment in which all states are Markov. This is what we want to solve. An MDP is a tuple $(S, A, P, R, \gamma)$, where $S$ is our state ...
2
votes
1answer
63 views
Why do I get the best policy before Q values converge using DQN?
I have implemented DQN algorithm and wonder why during testing, the best performance is achieved by a policy from about 300 episode, when mean Q values converge at about 800 episode?
Mean Q-values ...
2
votes
1answer
34 views
Is a learned policy, for a deterministic problem, trained in a supervised process, a stochastic policy?
If I trained a neural network with 4 outputs (one for each action: move down, up, left, and right) to move an agent through a grid (deterministic problem). The output of the neural network is a ...
1
vote
1answer
160 views
What is a “learned policy” in Q-learning?
I am completing an assignment at the moment. One of the assignment questions asks how you identified the learned policy and how you obtained it. The question is a reinforcement learning question, and ...
2
votes
0answers
21 views
Does there necessarily exist “dominated actions” in a MDP?
In a Markov Decision Process, is it possible that there exists no "dominated action"?
I define a dominated action the following way:
we say that $(s,a)$ is a dominated action, if $\forall \...
1
vote
0answers
13 views
How to obtain the policy in the form of a finite-state controller from the value function vectors over the belief space of the POMDP?
I was reading this paper by Hansen.
It says the following:
A correspondence between vectors and one-step policy choices plays an important role in this interpretation of a policy. Each vector in $\...
1
vote
0answers
62 views
What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?
In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods.
AlphaGo Deepmind RL model has beaten the best Go ...
3
votes
1answer
107 views
Are there some notions of distance between two policies?
I want to determine some distance between two policies $\pi_1 (a \mid s)$ and $\pi_2 (a \mid s)$, i.e. something like $\vert \vert \pi_1 (a \mid s) - \pi_2(a \mid s) \vert \vert$, where $\pi_i (a\mid ...
0
votes
1answer
52 views
How to use and update a shared/global policy between Reinforcement Learning Agents
I would be grateful for some guidance on a RL problem I am trying to solve where multiple RL agents use a common/global policy at the initial state of an episode in the RL Environment, and then update ...
5
votes
2answers
299 views
Given two optimal policies, is an affine combination of them also optimal?
If there are two different optimal policies $\pi_1, \pi_2$ in a reinforcement learning task, will the linear combination (or affine combination) of the two policies $\alpha \pi_1 + \beta \pi_2, \alpha ...
0
votes
0answers
43 views
Off-policy full-random training in easy-to-explore environment
Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe).
In those environments, using off-policy algorithm, is it a good ...
0
votes
0answers
61 views
Are there any reinforcement learning benchmarks where the optimal policy is known for each environment?
There are multiple reinforcement learning (RL) benchmarks (i.e. a set of environments where we can test our RL algorithms), for example, the DeepMind Control Suite. However, given that I am currently ...
2
votes
1answer
112 views
Why do we have two similar action selection strategies for UCB1?
In the literature, there are at least two action selection strategies associated with the UCB1's action selection strategy/policy. For example, in the paper Algorithms for the multi-armed bandit ...
1
vote
0answers
35 views
Dynamically adapting activation function
I am training a network through reinforcement learning. The policy network learns rotations, but depending on the actual input (state), the output of the network should be restricted to be in certain ...
2
votes
0answers
33 views
What's an example of a simple policy but a complex value function?
Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared ...
2
votes
1answer
51 views
Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?
I was looking at the Bellman equation, and I noticed a difference between the equations used in policy evaluation and value iteration.
In policy evaluation, there was the presence of $\pi(a \mid s)$, ...
2
votes
0answers
25 views
Do we assume the policy to be deterministic when proving the optimality?
In reinforcement learning, when we talk about the principle of optimality, do we assume the policy to be deterministic?
2
votes
1answer
67 views
Is a policy in reinforcement learning analogous to a field such as APF?
If a policy maps states to actions in reinforcement learning, then for a path planning with obstacles, can't we simply use Artificial Potential Field fields for path planning and model policy ...
2
votes
1answer
65 views
Is it common to have extreme policy's probabilities?
I have implemented several policy gradient algorithms (REINFORCE, A2C, and PPO) and am finding that the resultant policy's action probability distributions can be rather extreme. As a note, I have ...
3
votes
1answer
115 views
What is meant by “generate the data” in describing the difference between on-policy and off-policy?
From the book:
Sutton, Richard S.,Barto, Andrew G.. Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 100). The MIT Press. Kindle Edition. "
following is stated:
"...
1
vote
1answer
59 views
Do we need multiple parallel environments to train in batches an on-policy algorithm?
When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ...
1
vote
0answers
63 views
Does the off-policy evaluation work for non-stationary policies?
As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies?
For example, IS (importance sampling)-based estimators, such as weighted IS or doubly ...
2
votes
2answers
192 views
Why is the policy not a part of the MDP definition?
I'm reading an article on reinforcement learning, and I don't understand why the agent's policy $\pi$ is not part of definition of Markov Decision process(MDP):
Bu, Lucian, Robert Babu, and Bart De ...
3
votes
1answer
113 views
Why do RL implementations converge on one action?
I have seen this happening in implementations of state-of-the-art RL algorithms where the model converges to a single action over time after multiple training iterations. Are there some general ...
3
votes
2answers
101 views
How does the AlphaGo Zero policy decide what move to execute?
I was going through the AlphaGo Zero paper and I was trying to understand everything, but I just can't figure out this one formula:
$$
\pi(a \mid s_0) = \frac{N(s_0, a)^{\frac{1}{\tau}}}{\sum_b N(s_0,...
2
votes
3answers
59 views
How to estimate a behavior policy for off-policy learning based on data?
I have a dataset which includes states, actions, and reward. The dataset includes information on the transition, i.e., $p(r,s' \mid s,a)$.
Is there a way to estimate a behavior policy from this ...
3
votes
1answer
157 views
Why do value iteration and policy iteration obtain similar policies even though they have different value functions?
I am trying to implement value and policy iteration algorithms. My value function from policy iteration looks vastly different from the values from value iteration, but the policy obtained from both ...
2
votes
1answer
71 views
Why do I need an initial arbitrary policy to implement value iteration algorithm
I've been recently given an assignment based on Reinforcement Learning and I'm supposed to implement the value iteration algorithm in a grid environment.
The assignment:
My doubt is why do I even ...
2
votes
2answers
134 views
Is my understanding of the value function, Q function, policy, reward and return correct?
I'm a beginner in the RL field, and I would like to check that my understanding of certain RL concepts.
Value function: How good it is to be in a state S following policy π.
...
1
vote
2answers
54 views
Why does the policy $\pi$ affect the Q value?
From my understanding, the policy $\pi$ is basically how the agent acts (i.e. the actions it will take in each state).
However, I am confused about the Q value and how it is "affected" by a policy. ...
3
votes
0answers
43 views
Representation of state space, action space and reward system for Reinforcement Learning problem
I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without ...
0
votes
0answers
56 views
How to understand and visualize a trained RL agent's policy when the state space is high dimensional?
What are typical ways to understand and visualize a trained RL agent's policy when the state space is of high dimension (but not images)?
For example, suppose state and action are denoted by $s=(...
2
votes
0answers
46 views
Can the importance sampling estimator have a non-stationary behaviour policy even if the target policy is stationary?
The inverse propensity score (IPS) estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained in the paper Doubly Robust Policy Evaluation and Optimization.
...
2
votes
0answers
66 views
AlphaGo Zero: Does the policy head give a probability for every possible move?
If I understood correctly, the AlphaGo Zero network returns two values: a vector of logit probabilities p and a value v.
My question is: in this vector that it is outputted, do we have a probability ...
3
votes
1answer
252 views
Why does having a fixed policy change a Markov Decision Process to a Markov Reward Process?
If a policy is fixed, it is said that a Markov Decision Process (MDP) becomes a Markov Reward Process (MRP).
Why is this so? Aren't the transitions and rewards still parameterized by the action and ...
4
votes
2answers
89 views
Why is having low variance important in offline policy evaluation of reinforcement learning?
Intuitively, I understand that having an unbiased estimate of a policy is important because being biased just means that our estimate is distant from the truth value.
However, I don't understand ...
3
votes
1answer
127 views
Can someone please help me validate my MDP?
Problem Statement :
I have a system with four states - S1 through S4 where S1 is the beginning state and S4 is the end/terminal state. The next state is always better than the previous state i.e if ...
1
vote
0answers
43 views
Finding optimal Value function and Policy for an MDP
I am solving an RL MDP problem which is model based. I have an MDP which has four possible states S1-S4 and four different actions A1-A4, with S4 being terminal state and S1 is the beginning state. ...
1
vote
1answer
33 views
Do I need to store the policy for RL?
I am creating a zero-sum game with RL and wondered if I need to store the policy, or if there are other RL methods that produce similar results (consistently beating the human player) without the need ...
2
votes
1answer
47 views
If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy?
I wrote a simple implementation of Flappy Bird in Python, and now I'm trying to train an agent to play it at a reasonable skill level using TFLearn.
I feed the network an input vector of size 4:
...
4
votes
1answer
2k views
What is the difference between a stationary and a non-stationary policy?
In reinforcement learning, there are deterministic and non-deterministic (or stochastic) policies, but there are also stationary and non-stationary policies.
What is the difference between a ...
3
votes
3answers
3k views
What is the difference between a stochastic and a deterministic policy?
In reinforcement learning, there are the concepts of stochastic (or probabilistic) and deterministic policies. What is the difference between them?
2
votes
1answer
65 views
Possible inconsistency in the Policy Improvement equation
I came across this formula in Sutton And Barto: RL an Intro (2nd Edition) equation number 4.7 (page number 78).
If $\pi$ and $\pi'$ are deterministic policies and $q_\pi(s, \pi'(s)) \geq v_\pi(s)$ ...
1
vote
0answers
34 views
Measure grid-world environments difference for reinforcement learning
I'd like to measure the difference between 2 grid-worlds to determine the generalization capacity of my agent using tabular Q-learning.
Example (OpenAI Frozen Lake) :
SFFF
FHFH
FFFH
HFFG
and :
...
4
votes
1answer
380 views
A few questions regarding the difference between policy iteration and value iteration [closed]
The question already has some answer. But I am still finding it quite unclear (also does $\pi(s)$ here mean $q(s,a)$ ?):
The few things I do not understand are:
Why the difference between 2 ...
3
votes
2answers
218 views
What is the difference between a non-stationary policy and a state that stores time?
This question is related to What does "stationary" mean in the context of reinforcement learning?, but I have a more specific question to clarify the difference between a non-stationary ...
12
votes
3answers
2k views
Is the optimal policy always stochastic if the environment is also stochastic?
Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic?
Intuitively, if the environment is ...
4
votes
1answer
146 views
What is the relation between a policy which is the solution to a MDP and a policy like $\epsilon$-greedy?
In the context of reinforcement learning, a policy, $\pi$, is often defined as a function from the space of states, $\mathcal{S}$, to the space of actions, $\mathcal{A}$, that is, $\pi : \mathcal{S} \...
6
votes
1answer
120 views
How do I compute the variance of the return of an evaluation policy using two behaviour policies?
Suppose there is an evaluation policy called $\pi_{e}$ and there are two behavior policies $\pi_{b1}$ and $\pi_{b2}$. I know that it is possible to estimate the return of policy $\pi_{e}$ through ...
1
vote
0answers
41 views
Where does the expectation term in the derivative of the soft-max policy come from?
At slide 17 of the David Silver's series, the soft-max policy is defined as follows
$$
\pi_\theta(s, a) \propto e^{\phi(s, a)^T \theta}
$$
that is, the probability of an action $a$ (in state $s$) is ...
