Questions tagged [policies]
For questions related to policies (as defined in reinforcement learning or other AI sub-fields).
23
questions with no upvoted or accepted answers
6
votes
0
answers
99
views
Proof that there always exists a dominating policy in an MDP
I think that it is common knowledge that for any infinite horizon discounted MDP $(S, A, P, r, \gamma)$, there always exists a dominating policy $\pi$, i.e. a policy $\pi$ such that for all policies $\...
- 185
3
votes
0
answers
110
views
Representation of state space, action space and reward system for Reinforcement Learning problem
I am trying to solve the problem of an agent dynamically discovering(start with no information about the environment) the environment and to explore as much of the environment as possible without ...
3
votes
0
answers
70
views
Can the importance sampling estimator have a non-stationary behaviour policy even if the target policy is stationary?
The inverse propensity score (IPS) estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained in the paper Doubly Robust Policy Evaluation and Optimization.
...
- 227
2
votes
0
answers
54
views
Does there necessarily exist "dominated actions" in a MDP?
In a Markov Decision Process, is it possible that there exists no "dominated action"?
I define a dominated action the following way:
we say that $(s,a)$ is a dominated action, if $\forall \...
- 31
2
votes
0
answers
103
views
What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?
In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods.
AlphaGo Deepmind RL model has beaten the best Go ...
- 201
2
votes
0
answers
52
views
What are some other real-life examples of simple policies but complex value functions?
Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared ...
- 1,131
2
votes
0
answers
28
views
Do we assume the policy to be deterministic when proving the optimality?
In reinforcement learning, when we talk about the principle of optimality, do we assume the policy to be deterministic?
- 153
2
votes
0
answers
115
views
Does the off-policy evaluation work for non-stationary policies?
As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies?
For example, IS (importance sampling)-based estimators, such as weighted IS or doubly ...
- 227
2
votes
0
answers
282
views
AlphaGo Zero: Does the policy head give a probability for every possible move?
If I understood correctly, the AlphaGo Zero network returns two values: a vector of logit probabilities p and a value v.
My question is: in this vector that it is outputted, do we have a probability ...
- 265
1
vote
1
answer
88
views
What is the best distance measure between policies that are not probability distributions?
This question asks if there is a way to measure distance between policies that are in fact probability distributions.
In the case of continuous control with deterministic policies where they take a ...
- 111
1
vote
0
answers
33
views
How to allow RL systems to find better policies after code changes?
Suppose that in version 1 of a reinforcement-learning system an optimal policy $A$ got generated for executing a task. But, in a newer version 2 of that application (with new code changes), there ...
- 171
1
vote
0
answers
19
views
How to obtain the policy in the form of a finite-state controller from the value function vectors over the belief space of the POMDP?
I was reading this paper by Hansen.
It says the following:
A correspondence between vectors and one-step policy choices plays an important role in this interpretation of a policy. Each vector in $\...
- 221
1
vote
0
answers
87
views
Dynamically adapting activation function
I am training a network through reinforcement learning. The policy network learns rotations, but depending on the actual input (state), the output of the network should be restricted to be in certain ...
- 31
1
vote
0
answers
166
views
Finding optimal Value function and Policy for an MDP
I am solving an RL MDP problem which is model based. I have an MDP which has four possible states S1-S4 and four different actions A1-A4, with S4 being terminal state and S1 is the beginning state. ...
- 83
1
vote
0
answers
38
views
Measure grid-world environments difference for reinforcement learning
I'd like to measure the difference between 2 grid-worlds to determine the generalization capacity of my agent using tabular Q-learning.
Example (OpenAI Frozen Lake) :
SFFF
FHFH
FFFH
HFFG
and :
...
- 139
1
vote
0
answers
60
views
Where does the expectation term in the derivative of the soft-max policy come from?
At slide 17 of the David Silver's series, the soft-max policy is defined as follows
$$
\pi_\theta(s, a) \propto e^{\phi(s, a)^T \theta}
$$
that is, the probability of an action $a$ (in state $s$) is ...
- 253
0
votes
0
answers
14
views
Finding an optimal action score function for Multi-Armed Bandit Problem
Considering a multi-armed bandit problem where there are :
...
0
votes
0
answers
30
views
Define possible?
In Reinforcement Learning, policies are defined in terms of possible actions (see for instance page 58 of the book by Sutton et al.). So, is any action that an agent has in its repertoire always "...
0
votes
0
answers
75
views
Policy and Discount Factor
This question is similar to this question, however it has a different question.
I'm learning MDP's and I'd like to know if I'm doing these exercises correctly:
Consider the following MDP:
Suppose a ...
- 145
0
votes
0
answers
348
views
DQN learns to always choose the same action for all states
I have created an RL model that uses QBased policy with a neural network for estimating Q values.
My action space is of 27 actions, where each action is a 3 tuple where each value can be 1, 2 or 3. ...
0
votes
0
answers
53
views
Off-policy full-random training in easy-to-explore environment
Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe).
In those environments, using off-policy algorithm, is it a good ...
- 266
0
votes
0
answers
147
views
How to understand and visualize a trained RL agent's policy when the state space is high dimensional?
What are typical ways to understand and visualize a trained RL agent's policy when the state space is of high dimension (but not images)?
For example, suppose state and action are denoted by $s=(...
- 121
-1
votes
1
answer
236
views
Is my understanding correct regarding the difference between policy and plan?
I am confused regarding the difference between policy and plan in reinforcement learning. According to my understanding, when we calculate the value of state using Bellman equation in deterministic ...
- 111